A new benchmark says today's coding AI models still don't really understand what their own code does when it runs.
Researchers built SWE-Flux, a test set of 480 questions pulled from 12 real Python repositories, each one grounded in actual instrumented test executions rather than hand-written or LLM-graded answers. The questions probe whether models can trace control flow, loops, program state, dataflow, exceptions, and invariants as code runs, not just read it statically. They tested five LLMs on it. The best model got just 37% of questions right.
That gap matters because reading code and predicting its runtime behavior are different skills, and most existing benchmarks only test the former. Models did fine on contained problems like a single function's invariants or a simple loop, but fell apart on dataflow that crosses functions, precise tracking of program state, and aggregating results across a whole test suite, exactly the kind of reasoning debugging and code review demand. The researchers also showed their answer-generation pipeline can auto-produce harder variants of existing questions by perturbing inputs, successfully doing so for close to 90% of cases, which suggests today's low scores aren't just a fluke of stale test data.
Coding assistants get marketed on how well they write code. This benchmark is a reminder that writing code and knowing what it will actually do at runtime are still two different problems, and right now the models are much better at the first one.