No AI model tested could defuse a single bomb — a task millions of human players clear regularly.
Researchers introduced GPTNT, a benchmark built on the cooperative game Keep Talking and Nobody Explodes, to stress-test multimodal AI collaboration under conditions that existing evaluations skip. The setup mirrors the game's core mechanic: one agent sees the bomb but lacks the defusal manual; the other has the manual but cannot see or touch anything. Both must communicate in real time against a live countdown. The researchers tested a range of closed- and open-source models and found none could defuse even a single procedurally generated puzzle under real-time conditions.
Most AI benchmarks test capabilities in isolation — vision, reasoning, instruction-following — and then assume those skills compose cleanly when combined. GPTNT challenges that assumption by stacking time pressure, information asymmetry, and imperfect communication all at once. The gap between component performance and actual collaborative performance turns out to be significant.
The researchers identified specific failure modes: poor state tracking, slow action under pressure, trouble resolving ambiguous instructions, and an inability to recover from errors mid-run. Crucially, the benchmark is designed to distinguish between what a model figures out on the fly versus what it has memorized — the manual or the partner can each be withheld to isolate those contributions. Because GPTNT runs on the real game engine with procedural generation, it can't be solved once and retired; new puzzles keep coming, and an active modding community means the benchmark can grow harder as models improve. That's a meaningful design choice at a moment when popular AI benchmarks tend to get saturated within months of release.