Teaching an AI to write SQL is one thing; teaching it to know when it got the answer wrong is harder.
A new framework called GradeSQL trains what researchers call Outcome Reward Models — scoring functions that evaluate candidate SQL queries based on semantic correctness rather than surface-level signals. The current standard approach, known as Best-of-N sampling, picks the query that executes without errors or that appears most often across multiple attempts. GradeSQL instead labels candidates automatically using execution results, trains a verifier on those labels, and uses that verifier to pick the best query from a generated pool. No human annotation required. Tested on the BIRD and Spider benchmarks across several open-source language model families, the approach beat execution-based selection by up to 4.33 percentage points on BIRD and 2.10 points on Spider — with bigger gains on complex queries.
The gap matters because Text-to-SQL is one of the more credible enterprise use cases for language models right now — companies want to let non-technical staff query databases in plain English. A query that runs but returns the wrong rows is worse than useless; it looks right. Better verification at inference time is a more tractable fix than retraining the underlying model.
The code, datasets, and models are public, which is the right move for a paper making benchmark claims. That said, BIRD and Spider are well-worn targets; how this holds up against messier, real-world schemas is the question the benchmarks cannot answer.