A study of an AI receipt-reading system finds its standard error-check quietly fails almost half the time.
Researchers tested Claude Sonnet 5 on 13,859 fields pulled from 800 receipts in the CORD dataset, where the model got 49% of fields right. They identified three ways the standard accept/reject check breaks: treating fields from the same receipt as independent when they are not, reusing the same data to both calibrate and test the system, and duplicate confidence scores that collapse the decision threshold. The researchers organized fixes into a tiered framework, but even the most common fix, splitting data into a calibration set and a separate validation set, misses its own 10% error budget in 47.5% of repeated trials, per the paper's own numbers. A stricter method, using per-group statistical bounds with exact tail guarantees, does deliver a genuine certificate, but only by being so cautious it approves a small fraction of fields.
Document extraction quietly underpins invoicing, expense reports, and compliance pipelines, so a claimed error guarantee that fails nearly half the time is riskier than admitting there is no guarantee at all. It is the same gap that has tripped up confidence scoring elsewhere in machine learning: a bound that holds on average across many runs is not the same as a bound that holds for the one run in front of you, and most teams have no way to tell which they got.
A human-verified spot check on this particular run measured actual risk at just 1.3%, well inside the 10% cap - a good outcome, but by the paper's own math, more lucky than proven, since the same check misses its budget in roughly half of comparable trials.