AI tools that analyze your spreadsheets are guessing more than you think.
A new paper argues that LLM-based data-analysis agents are graded on the wrong thing. Most benchmarks only check whether the final answer is correct, skipping a step the researchers call Data Exploration: figuring out what tables actually live inside a messy workbook, what the columns mean, how sheets relate to each other, and where the data quality problems are. The researchers built two benchmarks to test this directly, including one based on a real multi-sheet Vitamin D study dataset, and scored systems on a structured artifact capturing tables, columns, relationships, and profiling signals rather than just the final answer. They found that strong LLMs and existing agents routinely miss this logical structure even after reading the file content.
This matters because it reframes a familiar failure mode. When an AI analysis tool gives a wrong number, the instinct is to blame its math or its code generation. This research says the error often happens earlier, when the model misreads what a spreadsheet even contains. The paper also reports that explicit support for this exploration step improves downstream accuracy, which points to a fixable design gap rather than a fundamental limit.
The more interesting proposal is treating that structured artifact as a checkpoint. Instead of asking a domain expert to audit a wall of generated code, you'd ask them to confirm the AI correctly identified the tables and relationships before any analysis runs. That is a much smaller, more checkable task, and it fits the broader pattern in AI tooling this year: less trust in end-to-end outputs, more inspectable intermediate steps. Whether spreadsheet tool vendors adopt anything like this remains to be seen, but the underlying critique, that benchmarks reward the right answer for the wrong reasons, is hard to dismiss.