A new benchmark called PROOF pokes holes in how confidently language models recite facts.
Researchers built PROOF by turning a frozen snapshot of Wikidata into 18,486 multiple-choice questions covering 11,779 facts across 101 classes, 392 properties, and 14 domains. Every question includes an "I don't know" option, and roughly 1,849 are deliberately unanswerable "no correct option" traps. The team ran 18 open-weight models through 166,374 prompts each, then reworded a subset of questions nine different ways and altered decoding settings to see if answers held up. They also tested what happens when a false answer gets planted directly in the question.
Base accuracy ranged from 6.58% to 57.59%. Only the weakest model actually landed below the 8.64% you'd expect from random guessing - the strongest model cleared that floor by a wide margin, so this isn't a story about models that don't know anything. It's a story about how unevenly they know it: every single model showed a 19 to 36 percentage-point swing in accuracy across domains, and neutral rewordings alone could shift scores by up to 26.5 points. Adversarial phrasing broke as much as 79.4% of previously correct answers, and confidence scores often stayed high even when the answer was wrong.
A single "factuality" number on a model card was never telling you the whole story. PROOF just makes the gaps easier to see, and harder to wave away as noise.