Swapping the AI model doing the grading changes the grade — even when the answers being graded stay identical.
Researchers tested what happens when you upgrade the model used to evaluate other AI outputs, a practice now common enough to have a name: LLM-as-judge. Across four judgment datasets, they compared two upgrade paths — scaling Qwen3 dense models from 1.7B to 32B parameters, and moving between MiniMax M2 and M2.7 API releases. The short version: judge upgrades are not interchangeable. Only the Qwen3 1.7B-to-4B step produced a consistent, reliable gain. Adjacent MiniMax releases did not. Bigger judges reduced position and verbosity bias but didn't eliminate it. Bringing in a "jury" of repeated samples helped little when those samples made correlated errors.
This matters because LLM-as-judge is now the default scoring mechanism for a wide range of AI benchmarks, safety evaluations, and product comparisons. If the evaluator model is quietly swapped between releases, scores become incomparable — which means a headline number showing improvement might reflect a different ruler, not a better model.
The researchers call for LLM-as-judge reports to ship with dataset slices, bias probes, error-dependence estimates, and protocol audit trails — the kind of methodology appendix that most leaderboard posts skip entirely.