A new benchmark tests whether AI coding agents know when to stop guessing and start asking.
Researchers built ReqClarifyBench, a set of 341 software tasks spanning game, miniapp, and web development. Each task starts with a deliberately vague user request, paired with a human-validated requirement document and code-evaluation artifacts for scoring. Instead of real users, the benchmark relies on LLM-based simulators built on personas drawn from real-world conversations, so the AI being tested has to clarify requirements through back-and-forth dialogue before writing any code. The team ran six LLMs through this process, scoring both how well they drew out clarifications and how good the resulting requirement documents were.
The results undercut a tidy assumption: asking better questions doesn't guarantee a better spec. Models that scored well on clarification effectiveness didn't necessarily produce higher-quality requirement documents, and performance swung significantly depending on which user persona they were talking to. Coding-agent harnesses, the scaffolding meant to make base models act more like autonomous developers, didn't improve document quality either - they performed about the same as the underlying LLM alone.
That last point matters more than it sounds. Repository-level coding tools are increasingly sold as agents that can take a one-line request and run with it, but this benchmark suggests the running-with-it part is still shaky - most effort is going into writing code, not figuring out what to write. Treat any AI dev tool that promises to turn vague asks into working software with a little more suspicion until it shows its clarification work.