A new benchmark study suggests the standard way of asking an AI model to turn a plain-English question into SQL is quietly wrong more often than it looks.
Researchers tested two approaches on an insurance industry benchmark called ACME. The first, a common direct approach, hands a language model the database schema and asks it to interpret the question and write the full SQL query itself. The second, called semantic path compilation, has the model only ground phrases and pick from a constrained set of question-specific options, while ordinary code handles the graph traversal, aggregation grain, and final SQL construction. Across 38 adjudicated questions run three times each, the compiled approach was correct on every single run for 37 questions (97.4%), compared with just 21 questions (55.3%) for the direct baseline. The baseline produced 29 wrong-but-executed queries and seven more flagged as coincidentally right for the wrong reason, out of 114 total runs; the compiled approach produced zero wrong-but-executed runs and one refusal.
That gap matters because the scariest failure mode in enterprise text-to-SQL isn't a query that errors out, it's one that runs fine and returns a plausible, wrong number, like joining through the wrong relationship or aggregating at the wrong grain. Moving that decision-making out of free-form generation and into governed, checkable steps appears to close off a lot of the ways that can happen silently.
The researchers are careful to frame this as a systems result, not proof that code-based compilation alone did the work, since the compiled approach also receives more structured schema information than the baseline ever sees. Worth remembering before anyone tries to sell "compilation" as a drop-in fix.