A new benchmark called WordPolo turns AI evaluation into a word-guessing game, and the scores are messier than any leaderboard suggests.
Researchers built WordPolo, a word-finding task where a model starts with zero information and must guess a hidden target word, receiving only a semantic-distance score after each guess (1 means correct, higher numbers mean further away). Success requires the model to interpret those scores and narrow its search over multiple turns, rather than pattern-match to a single answer. The team ran 1,500 puzzles through GPT-4.1, Llama 4, Claude 3.5 Haiku, Qwen 3, the reasoning models o4-mini and Deepseek-R1, human players, and a hand-built heuristic. Solve rates ranged wildly, from 4 percent to 62 percent, and the researchers added progression-based metrics that reward partial credit for narrowing in on the right answer, not just outright wins.
Plain accuracy numbers would have hidden most of the interesting behavior here. The analysis found reasoning models like o4-mini and Deepseek-R1 can be undone by both overthinking and underthinking a puzzle, while the models that actually solved puzzles well tended to search the way humans do - testing a guess, reading the feedback, and adjusting course.
It's a reminder that a model that aces a multiple-choice exam and a model that can reason its way through ambiguity are not automatically the same model.