Vision-language models can describe a room. They mostly cannot reason about what would change if you moved a chair.
Researchers introduced MindEdit-Bench, a benchmark built from 1,003 human-verified questions across 120 private indoor scenes captured as smartphone photo triplets. Six tasks test spatial reasoning at increasing difficulty. The first four probe whether a model can perceive and reproject what is already visible. The harder two — labeled L4 and L5 — ask models to predict the consequences of hypothetically moving or rotating an object, where the correct answer does not appear anywhere in the input images. Fifteen VLMs were tested. Task-wise mean accuracy landed between 8% and 31%. Human majority-vote accuracy on the same questions ran 81%–97%, leaving a gap of at least 39 percentage points on every single task and 53 percentage points pooled.
The gap matters because counterfactual spatial reasoning is not an exotic capability — it is the baseline for anything from robotic manipulation to augmented-reality furniture placement to accessibility planning. A model that can describe a room but cannot reason about moving a lamp in it is a lot less useful than the benchmarks we have been using would suggest. The 120 scenes were kept off public datasets deliberately, to reduce the chance that models had simply memorized similar training images.
The structured answer format — 8 to 24 labeled choices per question — also exposed specific failure modes: models showed weaker inference along the camera depth axis and defaulted to fallback guessing on the hardest visibility-editing cases. That level of diagnostic detail is more useful than a single accuracy number, and it hints that the deficiency is architectural, not just a matter of training more data on the same tasks.