Large language models can pass calculus problems — but a sweeping new survey suggests the math skills underneath may be shakier than the leaderboards imply.
Researchers reviewed roughly 120 peer-reviewed studies and preprints on mathematical reasoning in LLMs, building a unified taxonomy that separates pretraining data, fine-tuning resources, and evaluation benchmarks by reasoning complexity. They analyzed architectures and training strategies — including tool integration, verifier-guided reasoning, and parameter-efficient adaptation — then compared how well existing metrics actually measure what they claim to measure. The short answer: not well enough.
The gap between getting the right final answer and demonstrating sound step-by-step reasoning turns out to be significant. Current benchmarks tend to reward answer accuracy while leaving process-level verification largely unresolved — meaning a model can arrive at a correct result via flawed logic and still score well. That distinction matters enormously for any real-world deployment where the reasoning chain, not just the output, needs to be trustworthy.
The survey also flags benchmark contamination and generalization failures as persistent problems — a reminder that a model acing a published test set tells you less than it used to. Until the field agrees on evaluation methods that penalize reasoning shortcuts, leaderboard math scores are best read as marketing copy with footnotes.