A new paper offers a theoretically grounded way to monitor deployed language models for two common failure modes — without retraining or human review.
Researchers studied how a quantity called the dominant eigenvalue of the empirical Fisher Information Matrix, or FIM, behaves under two stresses: feeding a model inputs that stray from its training distribution, and compressing its weights to lower numerical precision (quantization). For out-of-distribution inputs, they prove under a curvature-monotonicity assumption that the eigenvalue rises detectably above a calibration baseline. For weight quantization, they derive a perturbation bound via Weyl's inequality showing the eigenvalue is, under mild conditions, strictly elevated compared to the unquantized case. Across twelve models and 1,080 test trajectories, the measured results tracked those predictions. One striking data point: on a 4-bit quantized model, the calibration threshold for their monitoring statistic landed roughly 244 times higher than a preliminary full-precision estimate.
The practical pitch is a runtime monitoring statistic, sigma_t, that can be computed continuously as a model serves requests. If it spikes, something changed — inputs drifted, or the model's precision was silently reduced. That kind of cheap, automatic signal is genuinely useful for teams running quantized models in production, where silent degradation is the norm and debugging is expensive.
The authors are candid that the 244x figure is a single empirical measurement, not a closed-form prediction, and they list deriving that magnitude analytically as an open problem — which at least makes clear where the theory ends and the number-watching begins.