Benchmarking an AI agent once tells you almost nothing.
Researchers have proposed a formal experimental design framework for evaluating how AI coding agents — specifically Codex and Claude Code — behave when tasked with open-ended data modeling and analysis. Because these agents are stochastic, meaning they can produce different outputs from the same input, a single benchmark run cannot reliably characterize their behavior. The framework treats each agent as a "stochastic model-discovery operator" and measures several responses across repeated trials: output quality, dollar cost, wall-clock time, and process complexity. Controlled variables include reasoning effort, task type, optimization metric, and training data composition. The researchers also built a utility-aligned decomposition to test whether pushing an agent toward higher reasoning effort actually moves it in a direction that improves quality relative to cost — or just burns money.
This matters because AI coding agents are increasingly deployed for real scientific and analytical work, not just boilerplate generation. If their outputs vary wildly run-to-run, any single evaluation is noise. A framework that quantifies that variability gives developers and researchers something to actually optimize against — and a way to compare agents on cost-efficiency, not just headline accuracy.
The testbed here is networked word-forming games, which is a narrow domain; whether these findings generalize to messier, real-world analytical tasks remains an open question. But the methodological contribution — treating agents as experimental subjects rather than deterministic tools — is overdue.