A new evaluation framework catches AI agents that quietly skip steps in a multi-step task while their final answer still looks fine.
ContractEval, introduced in a new paper, represents procedural instructions as "query-active obligations" - the specific checks, branches, dependencies, and invariants that should apply to a given request - then matches those obligations against what the agent actually did. That turns a vague sense that "something's off" into distinct, labeled failures: skipped steps, wrong branches, ordering errors, extra actions, broken invariants, or violated output contracts. On a controlled suite of audited procedural contracts, judges that only look at the final output, and judges that review the full execution trace, both missed many deliberately injected structural failures. ContractEval, working from gold-standard expected and observed graphs, caught and localized all of them. An LLM-based version of the extraction step preserved most of that accuracy but proved noticeably more sensitive to calibration.
That gap matters because LLM agents are increasingly asked to run processes, not just answer questions - filing tickets, applying refund policies, following incident runbooks - where a plausible output can mask a skipped check that nobody would notice by reading the answer alone. Grading what an agent said or did misses what it was actually obligated to do for that specific request, which is precisely the blind spot ContractEval is built to expose.
Worth noting: the researchers themselves call this a diagnostic, not a compliance guarantee, and admit the practical LLM-powered version is "calibration-sensitive" - a polite way of saying it can still be talked into missing things.