Researchers have a new way to choose which training examples to use during LLM fine-tuning — and it skips the reference models most competing methods depend on.
The technique, called UDS (Utility-Diversity Sampling), selects training batches dynamically rather than grinding through a full dataset. Most existing online batch selection methods score examples by predicted usefulness alone and ignore whether the selected examples are actually varied — producing batches that can be redundant or skewed. UDS tackles both problems at once. It uses the nuclear norm of the model's logits matrix, a linear-algebra measure that captures both data utility and within-batch diversity in a single pass. It then checks new samples against a lightweight memory buffer of recently seen examples to avoid inter-batch redundancy, all without extra backpropagation. No reference model, no validation set required.
The practical argument here is straightforward: fine-tuning a large model on every available example is slow and can amplify whatever biases were already in the data. The researchers report that UDS consistently outperformed other online batch selection methods across multiple benchmarks at various data budgets, and reduced total training time compared to full-dataset fine-tuning — the comparison that matters for teams working on constrained compute. Shaving training cost while maintaining or improving benchmark scores is the kind of result that moves from paper to pipeline.
Code is public. Whether the gains hold when the training data is noisier than a curated benchmark — which it almost always is — will be the real measure of whether UDS earns a slot in anyone's fine-tuning stack.