Text-to-SQL benchmarks have a blind spot: they test yesterday's SQL.
Researchers introduced Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases, built specifically to test whether language models can write SQL that calls large language model functions inline - think sentiment analysis or similarity search embedded directly in a query on Snowflake. Existing benchmarks only evaluate conventional SQL, so the field has had no signal on this capability at all. The team built the dataset using an agent-based pipeline that rewrote enterprise SQL tasks into AI-native form and ran each instance through a multi-round execution protocol across separate time windows to confirm the results were stable before publishing.
The accuracy numbers are a reality check. The strongest proprietary models hit 67-70% execution accuracy; the best open-source model reached 58.1%. The gap traces mostly to three failure modes: specifying predicates correctly, grounding queries to the right schema, and parameterizing the AI functions themselves. More striking is what the researchers found when they tested agent frameworks designed for traditional text-to-SQL - those elaborate pipelines for schema retrieval and table selection do not transfer here. A minimal agent setup matched or beat the more complex alternatives.
That last finding matters more than the accuracy numbers alone. The industry has spent significant effort building scaffolding around text-to-SQL agents, and this benchmark suggests that scaffolding was tuned to problems that AI-native SQL does not share. As cloud platforms like Snowflake bake LLM calls into SQL itself, the benchmark gap between what models can do and what analysts actually need widens - and the tooling built to close it may be solving the wrong problem.