A new benchmark that checks game code at every simulation tick found the best AI coding setup solved just over half of its 72 tasks.
Researchers built GameLogicBench, a set of 72 gameplay-logic tasks inside Godot game projects. Instead of scoring a recorded video or asking another model to judge the output, an automated evaluator checks whether the game's rules hold at every tick, across 403 hand-designed scenarios that expand into 1,451 test cases through seeded variations. To keep the grading honest, the evaluator also has to reject mutants, versions of a task with one required capability stripped out, while still accepting any correct implementation. Across 20 combinations of models and coding scaffolds, the best run solved 52.78% of the tasks, and under Claude Code, all twelve models tested solved progressively fewer tasks as the work moved from isolated mechanics to multi-system interactions to repository-scale features.
Most failed submissions still ran without crashing. They just got some required behavior wrong, the kind of bug a normal test suite or a quick playtest would miss. That is the real finding here: current coding agents can produce working looking code that quietly breaks the rules once a project outgrows a single mechanic, which is exactly the gap that matters for anyone considering agents for production game or software work.
The researchers also found that without validating the evaluator itself against those mutants, wrong answers slipped through as correct, and that agents given open network access sometimes just copied code from public repositories, a tidy reminder that a benchmark's headline number is only as good as the scaffolding checking it.