A new benchmark shows AI coding agents can write their own game-winning programs, with zero live model calls once play begins.
Researchers built a framework called Gauntlet that hands a general-purpose coding agent a bare setup: a game description, raw access to observations and actions, and an empty policy file - no algorithm, no strategy hints. The agent gets one live session to experiment and write a standalone controller, which is then frozen and tested on new game seeds while the model itself stays offline. On an unpublished procedural roguelike, success rates ranged from 0 to 86 percent, and every session from the newest model generation beat the best session from its predecessor. At full-game scale, one compiled controller beat every fair built-in AI opponent in StarCraft II plus two AI opponents given cheating advantages, and other single-session programs won complete games of Civilization (Freeciv) through total conquest on seeds they had never seen, though only at modest rates against the game's easier built-in opponents.
For years, feeding language models game knowledge didn't translate into competent play, even with memory modules, planners, and skill libraries bolted on. This result reframes the job: instead of narrating moves turn by turn, the model writes a program and hands it off, acting more like a compiler than a player. That distinction matters because it removes the latency, cost, and inconsistency of calling an LLM on every turn - the finished agent runs independent of the model that built it.
Beating StarCraft II's built-in bots and Freeciv's easier settings is a genuine first for this kind of standalone agent, but it's nowhere near AlphaStar-level play or a human opponent, and "modest win rates" against novice AI is a generous way to describe it.