ServiceNow Research Introduces EnterpriseOps-Gym: A High-Fidelity Benchmark Designed to Evaluate Agentic Planning in Realistic Enterprise Settings

Large language models (LLMs) are transitioning from conversational to autonomous agents capable of executing complex professional workflows. However, their deployment in enterprise environments remains limited by the lack of benchmarks that capture the specific challenges of professional settings: long-horizon planning, persistent state changes, and strict access protocols. To address this, researchers from ServiceNow Research, Mila and Universite de Montreal have introduced EnterpriseOps-Gym, a high-fidelity sandbox designed to evaluate agentic planning in realistic enterprise scenarios.

The Evaluation Environment

EnterpriseOps-Gym features a containerized Docker environment that simulates eight mission-critical enterprise domains:

• Operational Domains: Customer Service Management (CSM), Human Resources (HR), and IT Service Management (ITSM).
• Collaboration Domains: Email, Calendar, Teams, and Drive.
• Hybrid Domain: Cross-domain tasks requiring coordinated execution across multiple systems.

The benchmark comprises 164 relational database tables and 512 functional tools. With a mean foreign key degree of 1.7, the environment presents high relational density, forcing agents to navigate complex inter-table dependencies to maintain referential integrity. The benchmark includes 1,150 expert-curated tasks, with execution trajectories averaging 9 steps and reaching up to 34 steps.

Performance Results: A Capability Gap

The research team evaluated 14 frontier models using a pass@1 metric, where a task is successful only if all outcome-based SQL verifiers pass.

The results indicate that even state-of-the-art models fail to reach 40% reliability in these structured environments. Performance is strongly domain-dependent; models performed best on collaboration tools (Email, Teams) but dropped significantly in policy-heavy domains like ITSM (28.5%) and Hybrid (30.7%) workflows.

Planning vs. Execution

A critical finding of this research is that strategic planning, rather than tool invocation, is the primary performance bottleneck.

The research team conducted ‘Oracle’ experiments where agents were provided with human-authored plans. This intervention improved performance by 14-35 percentage points across all models. Strikingly, smaller models like Qwen3-4B became competitive with much larger models when strategic reasoning was externalized. Conversely, adding ‘distractor tools’ to simulate retrieval errors had a negligible impact on performance, further suggesting that tool discovery is not the binding constraint.

Failure Modes and Safety Concerns

The qualitative analysis revealed four recurring failure patterns:

1. Missing Prerequisite Lookup: Creating objects without querying necessary prerequisites, leading to “orphaned” records.
2. Cascading State Propagation: Failing to trigger follow-up actions required by system policies after a state change.
3. Incorrect ID Resolution: Passing unverified or guessed identifiers to tool calls.
4. Premature Completion Hallucination: Declaring a task finished before all required steps are executed.

Furthermore, agents struggle with safe refusal. The benchmark includes 30 infeasible tasks (e.g., requests violating access rules or involving inactive users). The best-performing model, GPT-5.2 (Low), correctly refused these tasks only 53.9% of the time. In professional settings, failing to refuse an unauthorized or impossible task can lead to corrupted database states and security risks.

Orchestration and Multi-Agent Systems (MAS)

The research team also evaluated whether more complex agent architectures could close the performance gap. While a Planner+Executor setup (where one model plans and another executes) yielded modest gains, more complex decomposition architectures often regressed performance. In domains like CSM and HR, tasks have strong sequential state dependencies; breaking these into sub-tasks for separate agents often disrupted the necessary context, leading to lower success rates than simple ReAct loops.

Economic Considerations: The Pareto Frontier

For deployment, the benchmark establishes a clear cost-performance tradeoff:

• Gemini-3-Flash represents the strongest practical tradeoff for closed-source models, offering 31.9% performance at a 90% lower cost than GPT-5 or Claude Sonnet 4.5.
• DeepSeek-V3.2 (High) and GPT-OSS-120B (High) are the dominant open-source options, offering approximately 24% performance at roughly $0.015 per task.
• Claude Opus 4.5 remains the benchmark for absolute reliability (37.4%) but at the highest cost of $0.36 per task.

Key Takeaways

• Benchmark Scale and Complexity: EnterpriseOps-Gym provides a high-fidelity evaluation environment featuring 164 relational database tables and 512 functional tools across eight enterprise domains.
• Significant Performance Gap: Current frontier models are not yet reliable for autonomous deployment; the top-performing model, Claude Opus 4.5, achieves only a 37.4% success rate.
• Planning as the Primary Bottleneck: Strategic reasoning is the binding constraint rather than tool execution, as providing agents with human-authored plans improves performance by 14 to 35 percentage points.
• Inadequate Safe Refusal: Models struggle to identify and refuse infeasible or policy-violating requests, with even the best-performing model cleanly abstaining only 53.9% of the time.
• Thinking Budget Limitations: While increasing test-time compute yields gains in some domains, performance plateaus in others, suggesting that more ‘thinking’ tokens cannot fully overcome fundamental gaps in policy understanding or domain knowledge.

Check out Paper, Codes and Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

The post ServiceNow Research Introduces EnterpriseOps-Gym: A High-Fidelity Benchmark Designed to Evaluate Agentic Planning in Realistic Enterprise Settings appeared first on MarkTechPost.

Credit: Source link