A research tool called HERMES proposes a smarter way to slice and sort the data that goes into pre-training large language models.
Most AI pre-training pipelines sort their data into categories first — by topic, format, or source — and then decide how much of each category to feed the model. The problem is those categories are fixed: change the resolution and you rebuild the whole label system from scratch. HERMES takes a different approach. It annotates each document once using a learned transform followed by three-stage residual vector quantization, producing a hierarchical code that can be read at different levels of detail — from broad buckets down to roughly 130,000 fine-grained cells — by simply changing how much of the code you look at. On a 1-billion-parameter model trained on 25 billion tokens, adjusting the prefix length and applying a quality-coverage mixing rule produced a measurable lift of about 0.025 across a 16-task capability benchmark. Go one level finer, though, and the same rule stops helping because the candidate pools shrink too much.
The finding matters because data mix design is one of the last big levers in model pre-training that labs still treat as art rather than science. HERMES makes it easier to run controlled experiments across granularity levels without rebuilding infrastructure each time — a reusable substrate rather than a one-shot label set. That could accelerate the kind of ablation work that separates genuinely better training recipes from lucky hyperparameter rolls.
The coarse-level clustering is roughly on par with standard KMeans methods, so HERMES is not claiming to out-cluster anyone — it is claiming the hierarchy is the point. Whether that argument holds at the scale that frontier labs actually train at remains untested.