A new study says transformers learn arithmetic long before they can show it.
Researchers trained encoder-decoder models on one-step Collatz prediction and watched grokking, the long gap between fitting the training set and suddenly generalizing. They found the encoder organizes parity and residue structure within the first few thousand training steps, while output accuracy sits near chance for tens of thousands more. Causal interventions point to the decoder as the bottleneck. Transplanting a trained encoder into a fresh model sped up grokking by 2.75 times, while transplanting a trained decoder actively hurt, and freezing a converged encoder to retrain only the decoder erased the plateau and reached 97.6% accuracy, against 86.1% for joint training.
Grokking has long been treated as a puzzle about when a model acquires structure. This work reframes it as a plumbing problem. The knowledge arrives early; the decoder simply cannot reach it, which makes the famous delay less about insight and more about access. It is the difference between a model that has not figured something out and one that has but cannot yet say it.
The twist is that access depends on how you write the numbers. Across 15 numeral bases, those whose factorization lines up with the Collatz map's arithmetic, like base 24, reached 99.8% accuracy. Binary failed completely, its representations collapsing and never recovering. The base acts as an inductive bias that controls how much local digit structure the decoder can exploit.
Put those two findings together and arithmetic generalization starts to look less like a test of whether a model understands math and more like a test of whether its input encoding hands the right structure to the part that has to produce the answer. That is a useful, slightly deflating result. Some of the sudden-insight drama in grokking may be an artifact of representation choices we control rather than emergent reasoning we stumbled onto. It is worth remembering the next time a model's late breakthrough gets sold as a spark of understanding.