A new reward model for vision-language AI needs zero human preference labels and still beats GPT-4o on benchmark tests.
Researchers took a frozen Qwen3-VL backbone and attached a tiny linear head to one of its intermediate layers. They trained that head to score an authentic web caption higher than a deliberately corrupted version of the same caption - no human ever rated which pairing was better. Labeled data was used for exactly one thing: picking which layer to tap, a choice worth 5.8 points of benchmark accuracy over using the final layer. At 32B parameters, the resulting head matched or beat GPT-4o on three benchmarks, including a 6.6-point lead on VL-RewardBench, using a few thousand parameters and roughly a thousandth of the compute fine-tuning would require.
Reward models typically need costly, inconsistent human preference labels and get graded by benchmark rank rather than real-world accuracy. This method replaces labels with a free synthetic signal and still wins - but the team also found that retraining the head on a policy's own verifier-labeled outputs cut caption hallucination and lifted image-generation success, even though it scored worse on every standard reward benchmark. That is a direct hint that current benchmarks may be optimizing for the wrong thing.
Two heads trained on different random perturbations tied on the public benchmark but disagreed by 6.4 points when judging actual hallucinations - a reminder that a matching score and a correct judgment are not the same claim.