A new benchmarking framework suggests AI coding models are better at pattern-matching than problem-solving.
Researchers have introduced AlgoBench, a framework that generates novel algorithm problems by systematically mutating known competitive-programming challenges. The twist: each new variant is designed so the original solution fails, forcing a model to actually reason rather than retrieve. Beyond the standard pass-rate metric, AlgoBench introduces five complexity-aware measures — OPTT, OPTS, TRAPRATE, GAPT, and CONSENS — to check whether a solution is not just functionally correct but also efficient enough for the problem at hand. Experiments across multiple large language models showed performance dropping sharply on these variants compared to the source problems.
This matters because benchmarks like HumanEval and LiveCodeBench have become compromised over time: problem statements, editorial write-ups, and model-generated solutions leak into training data, letting later models score higher without becoming smarter. AlgoBench's mutation approach offers a moving target that is harder to saturate by data exposure alone. The error analysis found that most failures were algorithmic, not syntactic — models knew how to write code, they just reached for the wrong algorithm.
It is a familiar dynamic in AI evaluation: the moment a benchmark goes public, it starts becoming a training signal, and the leaderboard stops measuring what it was built to measure. AlgoBench is not a permanent fix, but it is a more honest test of whether a model can adapt when the rules change.