Ask a chatbot a vague question and it will usually just guess - a new study shows exactly how bad that guessing is.
Researchers built MT-InfoSeek, a benchmark of 5,251 problems and 9,006 task instances spanning math, logic, biology, medicine, and general knowledge, designed to test whether language models know when a question is underspecified and ask the right follow-up. Each problem is built around k, the number of missing variables needed to pin down a unique answer. The team tracked three things: what models ask, when they ask it, and whether the answers they get actually resolve the question. Across the board, performance dropped as more information was missing, and models routinely underestimated how much they still needed to know - in logic problems requiring two extra facts, they lowballed the gap about four times more often than they overshot it.
The bigger problem is not just asking too few questions, it is asking them in the wrong order. When problems had dependencies, where one fact only makes sense once another is known, asking out of sequence hurt accuracy even after the model eventually gathered everything it needed. The researchers also found that telling a model the exact number of missing facts barely helped, which suggests the failure is not about counting, it is about actually reasoning through what is missing.
Most benchmarks only check whether the final answer is right, which lets a model's confident wrong turns hide behind a lucky guess. This one grades the interrogation, not just the verdict, a more honest way to catch models that answer first and think never.