A new paper shows how to measure a neural network's learning geometry without reconstructing the giant matrix that describes it.
The Neural Tangent Kernel (NTK) is a mathematical object that captures how a network's parameters respond to training updates at a given point in time. Computing it directly for any real-sized model is usually impossible, since the memory and compute costs explode. The researchers instead use randomized trace estimation, a technique called Hutch++, to approximate useful NTK statistics, including trace, Frobenius norm, effective rank, and alignment, using only matrix-vector products from standard automatic differentiation. They tested the approach on multilayer perceptrons, recurrent GRUs, and a 410-million-parameter language transformer, reporting orders-of-magnitude speedups over direct computation.
That speed matters because NTK theory has mostly lived in papers about tiny toy networks. It has been too expensive to check whether its predictions hold up at the sizes people actually train. With these estimators, the team used NTK alignment to study rich versus lazy training in RNNs and as a regularizer during data-scarce knowledge distillation, where it modestly improved generalization.
The gains are real but modest, and that is the honest framing here. This is not a new training method, it is a cheaper ruler. If it holds up outside the three architectures tested, it could let researchers finally test NTK-based theories of generalization on the models people actually deploy, instead of toy networks and a wave of hands.