A new study shows that minimal transformer circuits can perfectly solve a coreference-style reasoning task — and the results are simpler than most researchers expected.
Researchers trained small, attention-only transformers from scratch on a symbolic version of the Indirect Object Identification (IOI) task, a standard benchmark for testing whether a model can track which noun a pronoun refers back to. The surprise: a single-layer model with just two attention heads hit perfect accuracy, with no MLP layers and no normalization. Analyzing the residual stream revealed that the two heads divide the labor into additive and contrastive subcircuits that work together to resolve the task. A two-layer, single-head variant leaned on query-key interactions to pass information between layers.
The findings matter because mechanistic interpretability — the project of reverse-engineering what neural networks actually compute — usually has to fight through the noise of large pretrained models. By training small models specifically for one task, the researchers got circuits clean enough to inspect directly, offering a controlled environment for testing theories about how transformers reason. That kind of testbed is rare and genuinely useful for the field.
The catch is the usual one: a symbolic, lab-clean version of IOI is a long way from the ambiguous, context-heavy coreference that shows up in real text. Whether these minimal circuits scale or generalize remains an open question — but at least now there is a legible baseline to argue about.