A new filtering method can catch degrading AI training data without needing a model to check it.
Researchers studying "model collapse" - the way AI models lose vocabulary and start repeating phrases after repeated rounds of fine-tuning on their own generated text - tested a statistic called the Kontoyiannis entropy rate estimator. It is computed directly from raw text using match-length statistics, with no model involved at all. In a six-generation fine-tuning test on Llama-3.1-8B, filtering training data with this entropy measure increased unique trigrams by 42%, vocabulary by 30%, and cut repetition by 19%, all statistically significant results. A standard baseline that filters using a model's own log-probabilities showed no measurable improvement on any of those diversity metrics in the same test.
Model collapse is a growing concern as more of the text available for training is itself AI-generated, and most existing fixes assume you have a working model or clean human data on hand to check against. This method needs neither. Because it is just text statistics, it is cheap enough to run at scale, and it could help keep multi-agent systems from converging into the same repetitive outputs when no single trusted model is available to referee.
The researchers tested the approach across four domains, two temperatures, and two generator-scorer model pairs. That is still a narrow slice of the real world, but it beat a well-established baseline that could not clear statistical significance at all - a reminder that fancier does not automatically mean better.