A new study finds that renaming an option from "1" to "yes" can flip an AI decision model's answer, even when nothing else about the question changes.
Researchers tested Jev and two similar open-weight models across 1,200 workflow decisions, where each model picks between two predefined options tied to a written rubric. They kept the question, the underlying state, and the rubric wording identical, and changed only which option name, 0/1 or no/yes, was attached to which rubric. That single swap changed 70.4 more answers per hundred and collapsed the models' ranking ability, with AUC falling from .94 to .23. A hosted commercial model showed the same weakness, dropping from .8146 to .5806 AUC and flipping answers 24 times more often than its own repeat-test baseline. Swapping in neutral labels, or random character strings instead of words like yes and no, made the effect disappear without hurting accuracy.
This matters because these are exactly the systems software pipelines are built to trust: they always return a validly formatted decision, so a downstream script never sees an error. But a parseable answer says nothing about whether the model understood which option meant what. Throughout all these tests, the type-error rate stayed at zero percent even as decision accuracy fell apart.
Swap in loaded words like yes and no, and the model appears to react to the label's emotional charge rather than the rubric it is bound to. Anyone treating clean JSON output as proof that an AI judge is reasoning correctly should read this paper twice.