A new benchmark designed to separate thinking from memorization finds that AI "reasoning" is mostly just retrieval in disguise.
Researchers released IsoSci, a benchmark built around pairs of science problems that share identical logical structure but draw on different subject-matter knowledge. The idea: if a model truly reasons abstractly, it should perform equally well on both problems in a pair. Tested across five model pairs from four model families, the benchmark found that 91.3% of gains attributed to reasoning mode were knowledge-dependent, not structure-invariant. On highly capable models, switching reasoning on improved accuracy by less than 5 percentage points across all domains.
The sharpest finding involves o3-mini, a model explicitly specialized for reasoning. It beats its standard counterpart by 19.2 percentage points on GPQA Diamond - a widely cited reasoning benchmark - yet underperforms by 24.7 percentage points on IsoSci. That gap suggests benchmark selection isn't a neutral choice: it actively shapes what conclusions labs and researchers can draw about whether reasoning modes actually work.
Chain-of-thought reasoning has been one of the most hyped techniques in AI since it was popularized in 2022, with labs pointing to benchmark improvements as evidence that models are learning to think, not just pattern-match. IsoSci offers a controlled argument that the distinction may be murkier than the leaderboards imply - and that a model acing one test while failing a structurally equivalent one should make anyone cautious about what benchmarks are actually measuring.