A new paper offers a shortcut for getting neural networks to say how confident they actually are.
Researchers propose a low-rank generalized Laplace approximation for uncertainty quantification in neural networks, built from a small number of data-informed curvature directions rather than full Bayesian inference over every parameter. Starting from a generalized Bayesian posterior defined through an empirical loss, they construct a local Gaussian approximation around a pretrained set of weights within that active subspace. Variances inside the subspace come out in closed form, and the prior variance is calibrated with an empirical Bayes procedure. The authors also test two ways of scaling the posterior: the standard approach based on summed negative log likelihood, and a mean-loss version that normalizes the loss by the number of data points.
The scaling choice turns out to matter a lot. Standard scaling shrinks posterior variance in the leading directions as a function of data size, and in regression problems that forces the method to add extra weak-curvature directions just to match calibration data, which can make predictive intervals less coherent and drag the predictive mean away from the pretrained model. Mean-loss scaling avoids that: it keeps the active subspace smaller and more stable, and produces calibrated, coherent confidence intervals.
Full Bayesian deep learning has promised trustworthy uncertainty estimates for a decade without becoming the industry default, mostly because it is too slow and too expensive to run at scale. A cheap, low-rank stand-in is only interesting if it holds up outside a calibration benchmark.