A new paper argues that "grokking" is a fragile statistical artifact, not a reliable indicator of genuine learning.
Researchers studied Glimmer-1-Base, which the paper describes as a publicly released Llama-style transformer with approximately 11,856 parameters. That is small enough to enumerate every weight, every attention pattern, and the complete input-output map. Instead of reporting a single dramatic training run, the team measured grokking as a multi-seed rate across modular arithmetic tasks. The approach overturned three striking single-run results in their own dataset, each one a seed confound.
The most unsettling finding is numerical. Two changes to the floating-point environment, switching between CPU and GPU or varying the CPU thread count, each flipped a minority of same-seed outcomes without any detectable shift in the aggregate rate. That means published grokking results can quietly change depending on hardware, and the field would have no way of knowing because multi-seed reporting has not been standard. The coverage threshold that gates grokking tracks output cardinality, specifically the modulus in modular arithmetic, more than task structure.
Years of theorizing about late generalization may rest partly on results that were accidents of hardware configuration and lucky random seeds.