A research paper introduces a step-by-step scoring method for AI agents, challenging the field's reliance on final-answer benchmarks.
Researchers introduced Agent Step Value (ASV), a framework that scores each action an agent takes by measuring how that action shifts a language model evaluator's probability distribution over possible outcomes. Rather than collapsing an entire multi-step trace into a single success flag, ASV renders before-and-after snapshots of task state and asks a stateless LLM to score what changed. The team tested ASV on 100 open question-answering tasks using live PubMed data, evaluating 1,100 individual steps across 2,200 state pairs. They released the ASV Eval toolkit as a standalone open-source package alongside the paper.
Most agent benchmarks reward getting the right final answer, which says nothing about whether the path there was sensible or lucky. Step-level diagnostics let developers pinpoint the exact action where a multi-step chain went wrong, which matters when agents run automated workflows that no human is watching closely. As agentic systems take on longer, higher-stakes tasks, a binary pass-or-fail score is an increasingly poor diagnostic.
ASV depends on an LLM evaluator's log-probabilities, so its quality is bounded by whichever model you plug in - a limitation the broader "LLM-as-judge" literature has not solved yet.