New math says synthetic training data doesn't have to wreck your AI model - it depends entirely on when you feed it in.
Researchers studied one-pass stochastic gradient descent in high-dimensional linear regression to pin down exactly when synthetic data triggers "model collapse," the effect identified in 2024 research where any fixed share of synthetic data in training stalls performance no matter how much more data you add. They compared two setups: mixing synthetic and real data together throughout training, versus a two-stage approach that uses synthetic data only in an early phase before switching entirely to real data. Mixed training reliably produced collapse, and the effect got worse as models got larger. Two-stage training avoided the collapse floor altogether, and the team derived an exact condition for when synthetic pretraining actually beats training on real data alone, given the same real-data budget.
This matters because AI labs are running low on fresh human-generated text and leaning harder on synthetic data to keep scaling. This work turns a mostly anecdotal worry into a specific, provable claim: the danger isn't synthetic data itself, it's how you sequence it. It also complicates the assumption that bigger models are safer bets by default - under careless mixing, size amplifies the damage instead of absorbing it.
It's a theoretical result confined to linear regression, so no one should rewrite a production training pipeline off it tomorrow - but it's a rare case where "just order your data better" holds up under actual proof rather than vibes.