Researchers just showed that a small wording change can flip whether an AI model trusts its training data or the text you just handed it.
The study reproduces and stress-tests a 2023 paper by Yu et al. that examined how language models arbitrate between memorized facts and contradictory information placed in a prompt. The team reran the original world-capitals experiments across 31 models from the Pythia, GPT-2, Qwen3, and Ministral families, spanning base and post-trained variants, then extended testing to five more relation types from the ParaConflict dataset. Most of the original findings held up: larger models and more frequently seen entities lean toward memorized answers, though family-level variance is substantial. Some conclusions did not generalize cleanly, though. The entity-frequency effect disappeared entirely on Qwen3-14B and 32B, and post-training shifted the memory-context balance inconsistently across model families.
The detail worth sitting with is fragility. Question phrasing alone swung a model's reliance on memorized knowledge by up to 80 percentage points, and prose with no real relevance to the question could still pass as convincing supporting context. That is a headache for anyone building retrieval-augmented systems that assume a model will simply defer to whatever text gets injected into the prompt.
The industry's working assumption that context beats memorized knowledge is a rule of thumb, not a law, and this study shows it breaks in ways tied more to phrasing than to what a model actually knows.