Human feedback used to train AI may be systematically misleading, and a new paper spells out why.
Researchers examining reinforcement learning from human feedback (RLHF) found that the standard Bradley-Terry model — the statistical workhorse behind most preference-learning pipelines — conflates two very different reasons a comparison might be hard to make. A human rater might struggle to choose between two AI outputs because they are genuinely similar in quality, or because spotting the relevant difference requires attention the rater simply did not spend. The paper calls this "rational inattention" and shows the two causes cannot be separated from passive comparison data alone. A case study using Chatbot Arena voting data found a cyclic pattern in the comparisons — meaning no single reward score can consistently explain the rankings.
That matters because RLHF is the backbone of alignment for nearly every major language model, from GPT to Gemini to Claude. If the preference signal is noisier than the model assumes, the resulting reward function could encode evaluator inattention as genuine preference — quietly teaching the model to optimize for things that catch a distracted rater's eye rather than things humans actually value. A second case study found that response times and gaze data carry quality-gap information that the binary labels do not.
The authors stop short of proposing a production fix, but the implication is pointed: human feedback is not revealed preference, it is a measurement with a bandwidth limit. The field has spent years scaling the number of labels; this paper suggests the question worth asking is how much useful signal each label actually contains.