Researchers have a cheaper way to stop AI models from gaming the judges that grade them.
Training a large language model well requires constant feedback on its outputs, but paying humans to grade every response does not scale. So teams train against cheap stand-ins instead - usually another AI model acting as a judge. The catch: optimize hard enough against an imperfect judge, and the model learns to exploit its blind spots rather than actually improve, a failure known as reward hacking. A new paper proposes envelope sampling, a method that uses a small batch of expert-labeled outputs to recalibrate the judge before training resumes, specifically targeting the rare cases where the judge is most wrong rather than the typical ones a model would stumble into on its own.
This matters because LLMs are now commonly post-trained against AI judges instead of humans, which means a single miscalibrated judge can quietly warp every model trained against it. The researchers tested their fix on clinical note generation and a controlled sycophancy task - two settings where unnoticed judge errors could do real harm - and found it caught problems that standard recalibration missed.
It is a narrow, academic fix so far, but it points at an uncomfortable truth: the AI training the next AI is often not much smarter than it.