A new benchmark finds that even the best AI voice assistants sometimes approve car commands they should have refused.
Researchers built a 202-scenario test bed to evaluate how large language models make pre-action decisions in vehicle voice assistants - not just whether they execute a command, but whether they refuse, ask for clarification, require confirmation, hand off to manual control, trigger an emergency response, or do nothing at all. They scored five models against a seven-class taxonomy of correct responses. Alignment with the reference decision ranged from 40.1% for the open-weight Llama 3.2 3B up to 89.1% for Gemini 3.1 Pro Preview, with the three API-based models clustering between 83.2% and 89.1% and no statistically significant gap between them. Even the top performers made two to three False Execute errors - running a command that should have been blocked - across 161 scenarios that called for something other than execution, and the researchers found persistent, specific mistakes in decisions about when to require confirmation or defer to manual control.
That last detail matters more than the headline accuracy number. Confirmation and manual-control calls are exactly the judgment moments meant to catch a misheard or ambiguous command before it does something physical, like unlocking a door or shifting a gear, so errors there are where a benchmark score turns into a real incident. A follow-up test on Llama 3.2 3B showed that a structured authorization policy lifted its alignment to 40.1%, well above the 28.2-29.2% scored by schema-only or generic-safety prompting, but it still did not eliminate False Executes.
The paper's own conclusion is the least marketable part of the story: prompting alone, however well-structured, is not a safety system. Carmakers building on top of these models will need a separate rules layer that checks permissions and vehicle state before any command reaches the hardware - a less exciting pitch than "AI-powered voice control," but the one that actually keeps the doors locked.