A new paper finds little proof that a popular AI safety technique actually works.
The technique is called alignment midtraining, or AMT: instead of only fine-tuning a model at the end of training, labs feed it large volumes of alignment-relevant text earlier, during pretraining, hoping the behavior generalizes better once the model is deployed. Researchers tested the idea on models up to 110 billion parameters, using as much as 1 billion tokens of midtraining data. They found midtraining could nudge a model's underlying "motivation" in simple test cases. But that effect vanished the moment even a tiny fraction of later fine-tuning data hinted at a different motivation. In a separate test involving rule-following, models only reliably learned a rule if it showed up somewhere in the midtraining or post-training data - implicit generalization to unstated rules mostly didn't happen.
This matters because midtraining has been floated as a way to make alignment hold up in situations a model was never explicitly trained for - which is most of deployment. If a handful of contradictory examples in fine-tuning can undo it, and rules still need to be spelled out rather than inferred, midtraining looks less like a generalization breakthrough and more like another layer of pattern-matching with the same blind spots as standard fine-tuning.
The authors are blunt about the implication: they say there isn't enough public evidence to claim midtraining solves the core problem of aligning increasingly capable systems. For a field that sometimes talks about alignment techniques as though they're closer to solved than they are, a result this easy to break is worth sitting with.