Researchers have proposed a benchmark for a problem most agent evaluations skip: knowing when to interrupt a human.
A new research paper introduces JarvisBench, a test suite for an "attention-coordination layer" the authors call Jarvis, named after the fictional Iron Man assistant. The system sits between a person and one or more autonomous agents, deciding when an agent should keep working alone and when it needs a human's judgment. JarvisBench includes 45 task instances - 20 single-agent tasks and 25 workstreams grouped into 10 multi-agent projects - spanning 19 domains, selected and adapted from more than 2,000 public candidates. The tasks are built so the need for a human check-in surfaces naturally mid-task, not because someone left something out of the original prompt.
Most agent benchmarks score whether a task got finished, not whether the agent bothered the right person at the right moment - a gap that matters more as coding agents and browser agents run unattended for hours at a stretch. JarvisBench also grades the reverse direction: whether the coordination layer can answer a user's question about background work accurately and promptly without derailing the agent. That's a genuinely underexplored axis, closer to human-computer-interaction research than to a typical task-completion leaderboard.
The reference implementation ships with a full-duplex speech interface, a nice demo but also a reminder that a 45-task benchmark for a system named after a movie AI is a proof of concept, not an industry standard yet.