AI models that ace generalization benchmarks fall apart once the test actually demands reasoning.
Researchers built TranSGrid, a testbed that combines deductive, inductive, and abductive reasoning in a single task, to probe systematic generalization - the ability to solve new problems by recombining familiar pieces. They ran seven Transformer models on 4,800 TranSGrid instances. The best model solved 79.6% of a standard held-out test set but only 55.3% of TranSGrid, and just 15.8% of its hardest subset. The gap held even when problems stayed within the training data's length range, undercutting the common assumption that testing on longer sequences alone proves a model can generalize.
That matters because it suggests most existing generalization benchmarks are quietly easier than they look. They tend to simplify away either the inductive or the abductive reasoning demand, or both. When the researchers reintroduced those simplifications into TranSGrid one at a time - making actions compose in a roughly linear way, or making goals explicit rather than inferred - solve rates snapped right back to ordinary test-set levels.
It is the research equivalent of grading on a curve: strip out one variable and a stubborn benchmark suddenly looks solved.