A new benchmark shows that quantized language models often cannot keep straight whether a document in front of them is an instruction to carry out or text to summarize.
The benchmark, called IBBench-Light, feeds the same external record to a model twice: once framed as something to execute, once as something to read and process. Across twelve semantic bases, that produces 144 matched pairs per model, run on four quantized instruction models for 1,152 archived responses. The paired metric, PECA, only counts a pair as a success if both the execute version and the process version satisfy their output contracts. Qwen looked solid on each task alone, 132 out of 144 on execute prompts and 109 out of 144 on process prompts, but only 97 of those pairs succeeded together. A later rerun on Phi found that simply changing which tokens count as end-of-sequence markers shifted exact paired success from 0 out of 144 to 62 out of 144, with no retraining involved.
That gap between per-task scores and paired scores is the real finding. A model that looks reliable when judged one prompt at a time can still fail constantly when the same input has to be read two different ways. And a 62-point swing caused by a stopping-token setting means some of what passes for an instruction-following benchmark result is measuring the evaluation harness, not the model.
The researchers also ran a bounded comparison against IHEval using the same SmolLM2 checkpoint and output budget, keeping IHEval's own instruction roles and scorer intact. Worth remembering next time a model claims a high instruction-following score: ask what counted as the end of its sentence.