A research team has released a benchmark designed to expose what LLM-based diagnostic agents actually do wrong, not just whether they get the right answer.
The benchmark, called AgenticOpsEval, introduces two datasets — AIOps2025 and RCA100 — covering more than 500 failure cases drawn from two microservice systems: HipsterShop and the OpenTelemetry Demo Store. Faults span resource, network, runtime, middleware, database, and application-logic categories. Unlike most existing evaluations, which grade only the final root-cause answer, this one scores agents across three dimensions: where the fault occurred, what type it is, and whether the reasoning trace is grounded in relevant evidence. The datasets were labeled by domain experts and stress-tested through competitions involving more than 6,000 participating teams.
The distinction between outcome and process evaluation matters more than it sounds. An agent that lands on the right answer via lucky guessing is useless in production; an agent whose reasoning chain you can audit and trust is deployable. Benchmarks that ignore the middle steps have been quietly rewarding the former.
Most AIOps benchmarks to date have been narrow in fault coverage and thin on expert validation — this one is neither, which raises the bar for what a credible evaluation looks like. Whether the research community adopts it as a standard, or fragments into yet more competing benchmarks, is a different question.