Fine-tuning a language model apparently leaves fingerprints on words that have nothing to do with the task you trained it for.
A new arXiv paper introduces a technique called Active Taskless Distillation, or ATD, that transfers a model's post-training gains to a second model using nothing but single-word choices. Researchers picked prompts where a shared "ancestor" model was nearly 50/50 on which of two ordinary words to pick next, then trained a fresh copy of that ancestor purely on the small nudges a fine-tuned "teacher" version showed in those same word choices. No target-task examples, teacher logits, or access to the teacher's actual parameters were involved. In the main coding experiment, they took a coding-tuned Qwen2.5-1.5B teacher, extracted 5,664 of its single-word prompt responses, and used them alone to train a plain copy of the base model, which then scored 5.34 percentage points higher on the HumanEval+ coding benchmark than a tightly matched control built to strip out that signal while keeping everything else the same.
This builds on "subliminal learning" research that showed a model's traits could bleed into unrelated outputs, but those earlier studies leaned on large amounts of teacher-generated text to pull it off. Here, a single word per prompt is apparently enough, and the effect reportedly also shows up in scientific knowledge, common-sense reasoning, and reading comprehension across different model sizes and families, scaling with how strongly the teacher was originally trained.
If a single word can smuggle out real capability, anyone trying to audit what a fine-tuned model actually learned, or keep its training data private, has a harder problem than they thought.