A new benchmark measures coding agents the way people actually use them — through conversation, not just final code.
Researchers introduced SWE-Together, a multi-turn benchmark built from 11,260 recorded user-agent coding sessions. They filtered those down to 109 repository-level tasks where the original repository state was recoverable, user goals were clear, and outcomes were observable. To run experiments, they built a simulated user powered by a large language model that replays the original user's intent and pushes back when an agent stalls or goes off track. Agents are scored on two things: whether the final code is correct, and how many corrective nudges they needed along the way.
Most coding-agent benchmarks hand an agent a complete task description and grade the result — a setup that ignores everything that makes real coding assistance hard. SWE-Together's finding that stronger agents both succeed more often and require fewer interventions is intuitive, but now it's measurable, which matters for teams choosing between frontier models or deciding how much human hand-holding to budget for.
Benchmark scores have a way of becoming marketing material the moment a lab finds a favorable one; SWE-Together will face the same pressure, especially since its simulated user is itself an LLM and could be gamed by agents tuned against similar simulators.