A new benchmark built on Kaggle datasets shows that AI coding agents fall apart when they have to find the right data before writing a single line of code.
Researchers introduced CODA-BENCH, a 1,009-task evaluation suite that places AI agents inside a Linux sandbox populated with data from the Kaggle ecosystem. Each task environment contains an average of 980 files, forcing agents to navigate realistic, noisy file hierarchies before they can begin any analytical work. The suite spans 31 communities and is the first benchmark designed to evaluate code and data intelligence together rather than in isolation. When tested against advanced agents, the top performers cleared only 61.1% of tasks — a ceiling that reflects how poorly current systems integrate data discovery with code execution.
Most existing benchmarks grade agents on code generation or data analysis separately, which is a comfortable abstraction from how engineers actually work. Real development environments are messy: sprawling file systems, irrelevant noise, and no hand-holding about where the relevant dataset lives. That 38.9% failure rate suggests the agents being marketed as autonomous engineers are still quite dependent on a clean, well-labeled setup to perform reliably.
The Kaggle ecosystem makes a sensible sandbox — it is public, varied, and scales to hundreds of datasets — but Kaggle competitions still come with cleaner documentation than most production environments. If top agents score 61.1% here, the number in a genuine enterprise codebase is probably lower.