A new benchmark called GeoFitBench-LLM tests whether AI models can figure out what map data a question actually requires - and most of them can't reliably.
Researchers built the benchmark to probe a step that usually gets skipped in AI evaluations: before a language model can answer a geospatial question, like finding flood-prone neighborhoods near a river, it has to identify which datasets, supporting resources, and constraints are needed. GeoFitBench-LLM scores models on two fronts - whether they grasp how locations and time periods relate to each other, and whether they understand what a term like "near" or "urban" actually means in context. The authors ran seven models through minimal prompting and checked their answers against a fixed, provider-independent list of correct requirements. GPT-5.4 Thinking came out on top, but still only recovered the full, correct set of needed resources in 66.67% of requests.
That gap matters because most geo-analytic tools built on LLMs assume the model already knows what data it needs before it starts retrieving or reasoning. If a model guesses wrong about which dataset or constraint applies, the answer downstream can look confident while resting on the wrong foundation - a failure mode harder to catch than a model simply admitting it doesn't know.
A two-in-three success rate from the best model here is a useful reminder that "AI understands geography" is still mostly aspirational.