Most coding-agent benchmarks hand out a pass or fail and call it a day.
AgentLens, released as open source on July 9, flips that approach. Instead of reducing an agent run to a single binary outcome, it evaluates the entire trajectory: whether the agent followed instructions, used its tools sensibly, caught its own mistakes, and communicated clearly with the user. Each run is scored using a mix of formal verification — for cases where an objective check exists — and LLM-written trajectory reviews that explain, in plain language, why a score landed where it did. Side-by-side comparisons let teams pit one model version against another on the same task.
The practical pitch is that AgentLens is designed to slot into a nightly evaluation pipeline, making it usable as a regression detector, not just a leaderboard generator. That distinction matters: most benchmarks tell you which model wins; this one is meant to tell you when your own agent gets worse between releases.
The pass-fail benchmark is a known problem in AI evaluation — Goodhart's Law, roughly, applied to agent leaderboards. AgentLens is not the first attempt to fix it, but pairing automated checks with LLM-generated explanations is a more honest acknowledgment that human-facing behavior is hard to reduce to a number. Whether the LLM-written reviews are themselves reliable is a question the benchmark does not fully answer.