Language models can generate spreadsheets now, but generating a good one is harder than it looks.
Researchers have released SpreadsheetArena, a benchmark platform that evaluates how well large language models build spreadsheet workbooks from natural language prompts. The method is blind pairwise comparison: human evaluators see two model-generated spreadsheets side by side and pick the better one, without knowing which model made which. The team also released a dataset of prompts, generated workbooks, and preference votes for further study. The project is framed as a living arena, meaning evaluations continue as new models are submitted.
The interesting finding is not which model won. It is that what counts as a good spreadsheet shifts depending on the prompt. Style, structure, and formula choices that work for one use case fall flat for another. Worse, when domain experts evaluated spreadsheets built for finance prompts specifically, even the highest-ranked models failed to follow standard professional practices. That gap matters because spreadsheets are not decorative — they carry budgets, forecasts, and decisions.
Most LLM benchmarks reward fluent prose or correct multiple-choice answers. Spreadsheets are a different animal: the output has rigid structure, nested logic, and layout conventions that vary by industry. SpreadsheetArena is a reasonable attempt to measure that complexity, though a benchmark built on preference votes will always carry the biases of whoever is doing the voting — a limitation the researchers acknowledge implicitly by running expert evaluations separately.