Neural network research may have a reproducibility problem hiding inside one of its trendiest phenomena.
A new paper audits the measurement tools used to study "grokking" - the delayed generalization effect where a network suddenly learns a task long after it appears to have stopped improving. The researchers found that on modular arithmetic tasks, a network's internal representations keep compressing for tens of thousands of training steps after the model has already generalized. Reading compression metrics at the grokking transition overstates the converged value by 3-5x on a multilayer perceptron and by 1.3-1.5x on a transformer. The lag between the accuracy transition and the compression transition runs at least 10,000 steps - roughly the same order of magnitude as the time it took to grok in the first place.
This matters because a wave of interpretability research has used those early-read metrics to draw conclusions about what grokking reveals about how networks learn. If the measurement window is wrong, the conclusions built on it are shaky. The paper also found that adding LayerNorm to a transformer changes the fraction of compression completed at the grok step from 0.87 to 0.25 - a detail that could silently invalidate architecture comparisons across prior work.
The authors released both code and an audit toolkit, and note that their adversarial test suite caught a false-confidence bug in their own implementation before publication - a rare and useful admission. Whether the broader field goes back to check its work is a different question.