A new proof tightens the math on how many concepts a neural network can cram into fewer dimensions than it has neurons.
The paper reframes a problem called linear accessibility, essentially whether you can cleanly read out which concepts a network is representing when it uses overlapping ("superposed") internal codes, as a version of compressed sensing, the same math used to reconstruct signals from partial data. Assuming noise that behaves predictably (subgaussian noise, a bell-curve-like pattern) and a fixed set of active features, the authors derive bounds showing the dimensions a network needs grow linearly with the number of active features and the logarithm of total features, not with the square of that number as earlier worst-case estimates implied. They check the bounds against approximations across a range of settings to confirm they hold up beyond pure theory.
That is a meaningfully looser constraint. Sparse autoencoders and other interpretability tools assume a model's superposed features can be teased apart linearly, and a linear rather than quadratic dimension requirement means models may have more room than pessimists assumed to pack in distinguishable concepts without them blurring together.
It is still a bound derived from tidy statistical assumptions, not evidence that any specific model's messy real-world internals actually cooperate this neatly.