Getting the easy graph questions right doesn't mean an AI model understands the graph, according to a new benchmark.
The benchmark, called GraphDecide, targets what its authors call System One models: AI systems like Jev that select directly from a set of candidate answers instead of generating open-ended text. Researchers tested Jev and comparable choice-based models alongside standard language-model baselines across fourteen model-interface configurations. The tests combine structural task profiles, side-by-side comparisons of graph-only versus graph-plus-text inputs, and checks on whether a proposed answer is merely workable or actually correct. The code and results are public on GitHub.
The results puncture a common shortcut for judging these systems. Models that correctly identify which nodes connect to which don't reliably extend that into correct judgments about the graph's broader structure, and pairing a model with extra text alongside the graph doesn't consistently make it smarter. For anyone building agents that lean on graphs, such as routing, scheduling, or dependency checks, that gap between spotting a connection and reasoning about structure is the part that matters.
It is the same lesson language-model benchmarks have taught before: passing the easy check and understanding the problem are two different achievements.