A new benchmark targets the gap between AI benchmark scores and real-world competence.
Researchers introduced Blind-Spots-Bench, a 235-sample dataset built from questions collected by students in an AI course. The tasks look trivial on paper - things like manipulating a string or drawing a dog with five legs - but they consistently trip up modern AI systems. The team cleaned and annotated the raw questions with structured reference solutions, then ran an automated grading pipeline across a wide range of language, vision-language, and image-generation models, both open-weight and closed-source.
The results complicate the tidy story that benchmark leaderboards tell. Closed-source frontier models outperformed open-weight competitors by roughly 10 percentage points on these tasks, even when both groups scored comparably on established benchmarks - meaning the standard tests may be measuring polish, not capability. No single model dominated across all task types, and some tasks defeated every model tested.
The practical implication is that a model can ace MMLU or HumanEval and still fail at something a ten-year-old handles without thinking. Blind-Spots-Bench positions itself as a diagnostic stress test rather than a ranking tool - the point is to find specific cracks, not crown a winner.
The broader benchmark ecosystem has a reproducibility and saturation problem: once a dataset goes public, models get trained toward it. Sourcing questions from students who have no incentive to game any particular system is a small but sensible hedge against that dynamic.