Most teams running LLMs in production treat evaluation as a scoreboard: benchmark, rank, ship. EvalLoop is a paper arguing that framing leaves the most useful signal on the table.
Researchers tested the methodology on a sales intelligence briefing task using 10 models, 3 providers, and 18 metrics grouped into 5 quality dimensions. The key finding: 69% of hallucination failures traced back to prompt-induced interpretation errors — invisible when you look only at aggregate scores. A targeted prompt fix lifted the best model's overall score from 82.6% to 94.6%, with Content Accuracy jumping 16.8 percentage points and Synthesis Power rising 26.4. A prior configuration change made without that diagnosis? Zero impact.
The implication cuts against how most teams operate. Aggregate benchmark scores compress failure into a single number and hide where a system is actually breaking. Decomposing quality into orthogonal dimensions — and then classifying why outputs fail within weak dimensions — turns evaluation into a repair loop rather than a beauty contest. That distinction matters most when a system is already deployed and you need to know what to fix, not just whether to switch models.
The paper also reports a 94% reduction in human review burden by running a one-time blind review on a four-model finalist panel rather than reviewing every configuration. The artifacts — a playbook, agent spec, and template repo — are published for reuse, which is either genuine open-source generosity or a bid to establish a methodology before the benchmarking industry does it for you.