A new study finds that a single irrelevant sentence can wreck a language model's math homework, and now we know exactly where it goes wrong.
Researchers studied how large language models solve grade-school word problems and found the process breaks into four distinct stages: Schema Abstraction, Operation Planning, Operand Binding, and Computation. Each stage runs in its own identifiable band of layers inside the model. When the researchers inserted a distracting, irrelevant clause into a problem, causing the kind of failure that has long puzzled people who use these models for math, they traced the corruption to one specific stage: Operation Planning. They further identified a set of attention heads responsible for that stage and confirmed their role by testing cause and effect in both directions.
This matters because "AI is bad at word problems with distractors" has been a known but poorly explained embarrassment for years. Vague explanations blame "reasoning limits" in general. This work instead shows the failure is narrow and mechanical: three of four stages hold up fine, and only the operation-planning step, essentially deciding which arithmetic operation to apply, gets thrown off by noise. That is a fixable target, not a fundamental ceiling.
It is also a reminder that a model getting the right answer most of the time tells you little about how it is getting there. Interpretability work like this is what separates "the model seems to reason" from "here is the exact circuit that reasons, and here is where it snaps." Expect follow-up work trying to patch that specific stage rather than just throwing more parameters at the problem.