A new evaluation method builds an AI judge's scoring rubric from real pass/fail outcomes instead of guesswork, and it catches more of its own agent's failures.
A new paper introduces RubricForge, a technique for building the rubric an AI judge uses to grade another AI agent's trajectories. Rather than hand-writing criteria, the common approach in tools like G-Eval, or fine-tuning the judge model itself, RubricForge evolves a rubric against a small set of trajectories with known outcomes, then freezes it and applies it to new trajectories in a single model call with no access to the underlying environment. The researchers tested it with one frozen 7B model acting as both agent and judge, on tau-bench (173 labeled trajectories from 220 rollouts) and WebShop (160 trajectories). Because the rubric is plain text, every verdict traces back to a named criterion rather than a black-box score.
The gain is not in raw accuracy. RubricForge's overall agreement with ground truth was not statistically distinguishable from a generic G-Eval judge, and its score calibration was slightly worse. What improved was the false-pass rate: on tau-bench, RubricForge wrongly credited a failed trajectory as a success 11.5 percent of the time, versus 17.3 percent for the generic judge, a meaningful cut but not the halving the framing might suggest. It also ranked graded WebShop outcomes more faithfully, with a Spearman correlation of 0.410 versus 0.370 for the baseline.
That distinction is the whole point: a false pass ships a broken agent, a false fail just costs a retry, so for anyone grading agents with agents, under-failing beats simply scoring well on paper.