Success rate tells you whether an AI agent finished the job. It says nothing about whether the agent solved it the same way twice.
A new paper proposes the Behavioral Consistency Metric (BCM) to fill that gap. The method trains a model to predict task success from behavioral features of an agent's execution trace, then measures how similar an agent's strategy vectors are across attempts. Researchers ran it on roughly 9,000 trajectories from six language model agents doing software engineering tasks. The headline result: consistency within a single task and consistency across different tasks are separate properties that can pull apart. Some agents repeat themselves reliably when you rerun the same task, but have no stable approach once the task changes - and that split shows up even when task difficulty is held constant, and even between agents with near-identical success rates.
That matters because most agent benchmarks only reward the destination, not the route taken to get there. An agent that solves 80 percent of tickets but takes a different, unpredictable path each time is harder to debug, harder to trust, and harder to hand off to a team that needs to know what it will do next. BCM is a process-level check on that kind of hidden variability, and it also surfaces a persistent gap between frontier and open-source models that outcome-only metrics have been missing entirely.
Worth noting: the authors are upfront that BCM is a diagnostic with specific conditions for when it's meaningful, not a new leaderboard number to chase. That kind of restraint is rarer than it should be in agent-benchmark papers.