A research team has released NormWorlds-CF, a benchmark designed to catch language models that ace normative reasoning tests without understanding the underlying logic.
The system uses a deterministic solver — not another AI model — to verify answers, generate proof certificates, and flag when a model reaches the right verdict through faulty reasoning. The benchmark includes 270 rule-world families and 1,080 test pairs, and it stages diagnostics to separate surface accuracy from genuine comprehension. Training a model only on final answers produced perfect scores on answer tasks but zero on falsification tasks, exposing a gap that standard benchmarks would miss entirely. The researchers also introduced a new training method, MR-GRPO, that awards partial credit based on reasoning structure rather than just correct outputs.
This matters because most AI safety and alignment work assumes that a model getting the right answer is a reasonable proxy for a model that has internalized the right rule. NormWorlds-CF directly challenges that assumption, and does so with a solver that removes the circularity of using one AI to judge another. The finding that answer-only training scores zero on falsification is the kind of result that should make teams shipping rule-following agents nervous.
OOD transfer — whether any of this holds on rule worlds the model has never seen — remains unsolved, which is the part that would need to work before any of this moves from benchmark to deployment.