Combining specialized AI models into one generalist model just got a statistical upgrade.
A new paper from researchers on arXiv argues that current model merging techniques are built on a flawed assumption. Most existing methods rely on geometric properties of the parameter space — essentially averaging or interpolating between trained models. The new framework reframes merging as probabilistic inference using a product-of-experts setup, where each single-task model acts as an expert that votes on the final merged parameters. The key finding: existing methods implicitly assume the residuals between merged and task-specific models follow a light-tailed Gaussian distribution, but in practice those residuals are heavy-tailed.
That mismatch matters because light-tailed assumptions underweight extreme parameter updates — exactly the directions that may carry the most task-specific signal. The researchers swap in a Cauchy distribution, which handles heavy tails better and still admits a provably convergent inference procedure. Experiments across multiple tasks and architectures show meaningful gains over current baselines.
Model merging is appealing precisely because it sidesteps the cost of fine-tuning on new data — you get a multi-task model for nearly free if the math works. The wave of merge-heavy model releases on platforms like Hugging Face has outpaced the theory justifying them; this paper starts to close that gap. Whether Cauchy experts hold up across the full diversity of models practitioners actually merge in the wild is a question the benchmark suite here cannot fully answer.