A new academic benchmark shows that grading an AI coding agent purely on whether its shell command worked can hide exactly how it's failing.
Researchers built QuoteBench from 56 one-shot tasks drawn from 14 real incident families, testing agents that send Bash commands through code which serializes, wraps, and reparses the model's raw output before running it. They inserted one deliberately unescaped parser into that path and compared results with and without telling the model the boundary existed. Replaying the exact same generated reply through the flawed parser cut success rates by 55.4 to 73.2 percentage points across eight test configurations. Disclosing the parsing boundary to the model recovered 30.4 to 60.7 points in six of those configurations, and did almost nothing in the other two.
The headline finding is that a single matched-execution score, the number most benchmarks report, can mask this entirely. One configuration studied, which the paper labels GPT-5.6-sol - a designation specific to this study, not a publicly documented release we could independently verify - showed a near-flat overall gap of just -3.6 points. That number was quietly averaging out 64.3 points of damage from the broken parser against 60.7 points of recovery once the boundary was disclosed. Two identical-looking scores, two completely different agents underneath.
The paper also found that swapping the deployment plumbing, not the model itself, can flip which agent wins a head-to-head comparison in at least one clear case among 26 pairs tested. That's a quiet indictment of leaderboards that report a single number without saying how commands actually got from model to shell.