Three popular open-source LLMs largely refused to cave when users pushed back on settled science - but the mechanisms underneath suggest the robustness may be accidental.
Researchers tested Llama-3.1-8B, Qwen2.5-7B, and Mistral-7B on climate change, vaccines, and evolution, applying skeptical pressure across single and multi-turn conversations. Rather than a uniform response, the models showed three distinct patterns: Llama doubled down, asserting scientific consensus more forcefully under pressure; Qwen softened its tone while holding the position; and Mistral mostly stopped responding altogether. Pairwise human judgments confirmed Llama's shift was a genuine stance change, not just a stylistic tweak. Activation analysis found the divergence concentrated in the middle layers of each model.
The critical finding is the gap between "robust" and "reliably robust." A model that ignores skeptical signals because it never registered them in the first place is not the same as one that understood the challenge and held firm. The researchers call this the difference between active and accidental robustness - and the distinction matters because accidental robustness is fragile. In the vaccine domain specifically, myth-rebuttal weakened under pressure, suggesting the behavior does not transfer cleanly across topics.
This is a useful check on optimism about LLMs as information sources. The models did not sycophantically flip to false balance, which is good. But behavioral tests alone cannot tell you why - and a model that appears resistant today may fold on the next sensitive topic just because it happens to perceive the skeptical signal differently there.