LLMs used for text-to-SQL translation have 27 documented failure modes — and the tools meant to fix them are quietly making things worse.
Researchers published the first systematic study of errors produced by large language models doing text-to-SQL translation via in-context learning — the technique where a model is shown schema examples and sample queries rather than fine-tuned on a specific database. The team tested four ICL-based systems across two standard benchmarks and catalogued 27 distinct error types grouped into seven categories. They then audited five existing repair methods and found that the fixes are computationally expensive, correct relatively little, and frequently introduce new errors of their own. As an alternative, the team built MapleDoctor, a targeted detection-and-repair framework that fixes 13.8% more queries than current approaches, cuts repair latency by 67.4%, and produces far fewer mis-repairs.
Text-to-SQL is embedded in a growing stack of enterprise software — analytics copilots, BI dashboards, self-serve data tools — where a query that returns wrong results often looks identical to one that works. Unlike a broken code path that throws an exception, bad SQL tends to fail silently with plausible-looking numbers. The 27-category taxonomy alone is a real contribution: the field previously had no shared vocabulary for what goes wrong, which made systematic improvement nearly impossible.
MapleDoctor's code is publicly available on GitHub; whether it makes it into production tooling is a separate question — the history of benchmark-beating AI research is full of papers that never left the lab.