A new benchmark says a chatbot that eventually books your ride can still be a pain to deal with along the way.
Researchers introduced RideWay, a benchmark that tests AI agents in a simulated ridehailing environment stocked with tools like search and booking calls. Alongside it comes a scoring method called Efficiency Utility, which only rewards trajectories that actually finish the task, then docks points for extra tool calls and extra back-and-forth with the user beyond a task-specific reference amount. The penalties aren't arbitrary: they're calibrated against human preference judgments, tested across 58 tasks and 24 models. The headline finding is that people penalize an agent for one too many questions about twice as harshly as they penalize it for one too many tool calls.
That asymmetry matters because most agent benchmarks still treat task completion as the whole story, which hides the difference between an agent that quietly double-checks a detail and one that keeps interrupting you to ask what you already told it. Efficiency Utility gets human-agreement accuracy up to 78.7% overall and 90.6% when agents differ mainly in how many turns they use, but it drops to chance level when trajectories differ only in tool-call counts, exactly the axis where the human annotators themselves agreed least.
That last gap is the honest part of this paper: it built a decent proxy for cutting excess back-and-forth, but admits that not all extra tool use is waste, and nobody, human or metric, can reliably tell which extra searches are worth it.