Two AI chat sessions can earn identical quality scores and still differ by 70 times in how much effort it took to get there.
A new framework proposed by researchers measures human-AI collaboration by pairing outcome quality with interaction cost, not quality alone. Testing across two datasets and four tasks, the team found sessions rated equally good could vary by up to 70x in the back-and-forth needed to finish. The tradeoff also shifts by task: some jobs reward users who keep iterating with the AI, while others go better with fast, minimal exchanges. Subjective satisfaction ratings from users turned out to be unreliable stand-ins for actual productivity.
Most AI evaluations still stop at whether the output was good, treating the conversation that produced it as a black box. This framework makes the hidden cost visible, and it found a clear pattern: productive sessions happen when the AI asks clarifying questions early and the user spends less time repairing misunderstandings later.
A five-star rating can still describe a slog. That gap between satisfied and efficient is exactly the kind of thing a product demo never shows you.