Researchers say the standard Transformer architecture has been quietly doubling up on duties, and untangling them makes models learn faster and perform better.
A paper posted to arXiv proposes what its authors call the state-prediction separation hypothesis: the single forward computation stream in a standard Transformer is simultaneously predicting the next token and storing state information for future predictions. The authors designed a Transformer variant with two separate computation streams, one for each role, then ran pretraining experiments at multiple scales. The results showed consistent improvements in validation loss and an average 2-3 percentage point gain on downstream tasks over standard Transformers.
Efficiency gains at the architecture level matter more than they used to. As pretraining costs stretch into the hundreds of millions of dollars, squeezing better performance from the same compute budget is exactly the kind of research that draws attention from labs under pressure to justify their spending. A structural fix that works across scales is harder to dismiss than a one-off tuning trick.
The authors also ran empirical analysis to rule out confounders and argue the gradient differences are fundamental, not incidental. That is the right thing to do, and worth noting, because a lot of architecture papers skip that step. Whether the finding holds at the scale frontier - or whether it has already been discovered and quietly shelved inside a large lab - remains, as usual, unknown.