Telling a language model how many errors to find turns out to be a cheat code for its benchmark score.
Researchers tested six large language models across 4,290 responses drawn from 143 writing passages, using a protocol they call ErrorBench. The setup varied only the prompt framing — specifically, whether the model was given a pre-populated error count before it started looking. Under one standard scoring method, that framing shift alone produced up to 0.79 points of F1 score gain. Under stricter matching, the inflation reached 0.96. A follow-up replication using the ERRANT 3.0.0 pipeline found that switching from a "blind" prompt to an anchored one raised the count-based F1 score by an average of +0.21 across six models — while the span-aware metric that actually tracks whether the model found the right words moved by only +0.04.
This matters because F1 score is the standard shorthand for how good a model is at catching errors, and the gap between the two metrics means published benchmarks may be grading on a curve that doesn't exist. If a model is simply pattern-matching on the number of errors it was told to expect, that skill does not transfer to real document review, where no one hands you the answer key first.
The study also found a behavioral split: GPT and Claude models were more instruction-compliant and produced larger count responses under the anchored prompts, while Gemini models showed smaller shifts. The practical recommendation is straightforward — strip error counts from evaluation prompts and always report span-level metrics alongside count-based ones. Whether benchmark authors will bother is a separate question.