Researchers have released ClassicLogic, a benchmark that puts AI agents through Sudoku, KenKen, Kakuro, and Futoshiki to test a capability most models still lack.
ClassicLogic is not a leaderboard for puzzle speed. Its core design is a hierarchical knowledge base for each game, where complex solving strategies are formally defined as compositions of simpler ones. That structure lets evaluators see exactly where an agent breaks down — whether it fails at basic rules or collapses when it needs to chain multiple strategies together. Puzzle difficulty is mathematically validated, not hand-tuned, which makes the benchmark harder to game than many existing tests.
Most AI benchmarks that test reasoning lean on language tasks, where it is genuinely difficult to separate compositional understanding from statistical pattern-matching over text. Puzzles with explicit, formal rules offer a cleaner signal. If a model can solve a Futoshiki grid it has never seen by combining strategies it learned separately, that is meaningful evidence of generalization — not just a good memory.
The benchmark is open-source and aimed squarely at neuro-symbolic systems, a class of AI that tries to marry learned representations with explicit logical rules. That field has spent years arguing it should outperform pure neural approaches on exactly this kind of structured reasoning; ClassicLogic gives it a proper place to prove the point — or not.