Teaching an AI model to follow instructions in its prompt can also teach it to believe whatever that prompt says, even when it is nonsense.
Researchers built FakeContextBench, a test set loaded with pseudoscientific claims across seven fields, to check whether large language models can tell when context in a prompt should actually override what they already know. Testing both commercial and open-source models, they found that pre-training alone does not give models this judgment. Worse, the standard fine-tuning methods used to sharpen in-context learning made things worse, cutting "reality accuracy" by as much as 14.95 percentage points compared to the untuned base model. The team's fix, called Jurisdiction In-Context Learning (J-ICL), bakes context validation directly into training. Across four model backbones, it lifted benchmark scores by an average of 5.84 points and reality accuracy by 9.20 points, and beat two established tuning methods, MetaICL and Symbol Tuning, by 18.09 points on a metric the authors call Reality Rate.
In-context learning is the mechanism behind most real-world LLM use: feeding a model documents, data, or instructions and trusting it to reason from them. This paper argues that the industry has been optimizing the wrong half of that skill, training models to extract patterns from examples while ignoring whether those examples deserve to be believed at all. That gap matters anywhere a prompt might contain fabricated statistics, invented citations, or confidently wrong claims, which is to say nearly everywhere.
It is a useful reminder that a model getting better at following your prompt is not the same as a model getting better at knowing when your prompt is lying to it.