A new study pokes a hole in the idea that self-driving AI can explain its own decisions.
Researchers built DriveMind, a large driving dataset derived from the nuPlan benchmark, pairing sensor data with chain-of-thought explanations aligned to the model's actual trajectory plan. They used it to train vision-language driving agents with two standard methods, supervised fine-tuning and group relative policy optimization, then tested what happens when key inputs are stripped away. Removing basic priors - the car's current position and navigation goal - tanked planning accuracy. Removing the model's own written reasoning barely changed the results. A closer look at where the models focus their attention confirmed it: the system was leaning on raw priors, not the chain-of-thought it generated.
That's a problem for the whole pitch behind these models. Vision-language driving agents are sold as more transparent than black-box systems because they narrate their reasoning before predicting a route. If that narration isn't actually driving the decision, it's closer to a plausible-sounding caption than a causal account - which matters a lot if anyone tries to use that reasoning to debug a crash or certify a system as safe.
The researchers call this the Reasoning-Planning Decoupling Hypothesis, and they're releasing a training-free probe so other teams can check their own models for the same flaw. Worth remembering next time a demo video shows an AI calmly explaining why it's changing lanes - it might just be talking to itself.