A new method lets you fine-tune a language model on math or code without quietly wrecking its common sense.
Researchers studied parameter-efficient fine-tuning, the standard way to teach a large language model a narrow skill without retraining the whole thing. They found that tuning different layers of a Transformer produces very different results: some layers boost the target task with minimal collateral damage, others wreck general reasoning ability. To find the good layers without expensive calculations, they used a cheap stand-in measurement, the similarity between a layer's input and output, which reliably flagged the layers most sensitive to adaptation. Adapters placed only on those layers, a method they call Layer-Selective LoRA, beat standard all-layer tuning on math and code tasks while keeping far more of the model's original commonsense reasoning intact.
Most fixes for this trade-off involve replaying old training data or adding regularization terms, both of which cost extra compute or data. This result suggests a cheaper lever: just be selective about where you touch the model. For any team running frequent fine-tunes on a shared base model, that is a meaningful efficiency gain, not a marginal one.
It will not shock anyone who has watched a specialist hire struggle with basic tasks outside their lane - the fix, it turns out, is training only the parts of the brain built for change.