AI/ ai · ai-agents · benchmarks · open-source

AI Agent Harness Beats Pricier Model For $15, Not $574

A new open-source runtime called StateM pushed a $15 AI agent run past the score of a $574 run using a pricier frontier model on a coding benchmark.

A new agent runtime called StateM gets language models to solve more of a demanding coding benchmark by changing how they manage state, not by changing the model.

StateM organizes long agent runs around durable states, phase-local context, checked transitions, and versioned procedures instead of leaving each step to the model's own memory. On Terminal-Bench 2.1, it pushes GPT-5.5 xhigh's score to 92.1%, ahead of the 91.9% posted by the pricier GPT-5.6 Sol Ultra reference run, and the same runbook transferred to GPT-5.6 without modification. With GPT-5.6 Sol xhigh, StateM reaches 95.3% raw accuracy across 445 trials and clears all 89 tasks at least once. A similar frozen setup lifts GPT-5.6 Luna from 76.7% to 85.4%, and under $38 of tuning raises DeepSeek-V4 Flash from 82.7% to 88.1%.

The number that matters more than the accuracy bump is cost. StateM's winning run cost about $15 in API usage, versus $574.68 for the GPT-5.6 Sol Ultra reference it beat. That gap suggests the scaffolding around a model can matter as much as which model you buy, especially for teams priced out of frontier-tier agent runs.

On a benchmark built to separate models that can actually plan long tasks from ones that only look like they can, a cheaper model with better bookkeeping just outscored the expensive one.

TR

The Revision

Written by an AI system from the public sources credited above. How we write →