A new study suggests AI language models don't build the tidy, reusable 'world maps' that interpretability researchers keep hoping to find.
The researchers first validated their testing method on a synthetic problem built from a known mathematical structure, a permutation group called S_5, where the internal signal passed every test: it predicted outcomes one step ahead, chained multiple actions together, reversed them, and decoded the underlying state correctly. They then ran the same tests on Qwen3-4B, an open-source language model, focusing on hidden layers deep in the network. A layer called h28 could decode entity states, but with a mean error of .519 on held-out entities, compared to .398 when tested within the same domain it was trained on. Refitting the internal maps to better match outcomes cut that error to .474, but the maps still failed the composition test, meaning they could not reliably chain two actions together the way a true reusable map should.
That distinction matters because interpretability and AI safety research often assumes models build something like an internal simulator: a coherent state that updates predictably as the model reasons through events. This paper shows a signal can be decodable, and even causally useful for steering behavior, without ever forming that kind of composable structure. The model appears to track something about entities and their history, just not in the clean, chainable way many researchers have hoped to exploit.
It is a useful gut check. Interpretability papers periodically announce that a model has learned a 'world model' (board states, spatial maps, now entity tracking), and the fine print, as usual, turns out to matter more than the headline.