A new benchmark framework exposes a gap that text benchmarks tend to obscure: large language models struggle to maintain a working map of their environment.
Researchers introduced AGI Maze, a grid-based maze framework designed to test whether agents can form persistent, structured representations of a partially observable world — not just pattern-match against what they can currently see. The framework offers a clean API and multiple difficulty levels, and the initial evaluation found that vanilla LLMs fail to represent maze state internally at inference time. A baseline agent that could use its message history as a working memory did better, but still could not reliably solve even small mazes within a step budget that a human would clear with ease.
The finding matters because it draws a line between two kinds of AI competence that often get conflated. Doing well on a reading-comprehension or coding benchmark does not require holding a mental model of a changing world — the answer is usually in the text. Navigation, planning, and most real-world tasks are not. AGI Maze is specifically engineered so that local pattern-matching cannot substitute for genuine world-modeling, which makes it a sharper diagnostic than most existing evals.
The benchmark joins a growing list of frameworks — including NetHack, BabyAI, and various embodied-agent environments — aimed at stress-testing the gap between fluent text generation and actual reasoning. The difference here is the deliberate focus on low-dimensional, grid-based inputs, which rules out the excuse that the model lacked sensory bandwidth and isolates world-modeling as the variable under test.