Popular text-to-image models score well on existing tests yet routinely ignore parts of what users actually ask for.
Researchers introduced Arena-T2I Hard, a 310-prompt benchmark built from real user requests logged in arena-style model competitions. Unlike prior benchmarks that test one instruction at a time, each prompt here carries roughly 30 yes/no constraints across six categories — including how well models render text inside images. The best closed-source system tested hit a score of 0.855, but the gap between the top and bottom of 11 evaluated systems stretched 33 percentage points, showing real differences that older benchmarks obscured. Critically, a model's ranking on public arena leaderboards had little bearing on how well it followed detailed prompts — those popularity contests reward aesthetics, not accuracy.
That split matters because creative workflows are not simple. A designer asking for "a vintage poster with bold red serif text, a bicycle in the lower left, and a mountain range at dusk" is not submitting an atomic instruction — and a model that gets the mood right while scrambling the layout is not actually useful. The researchers also propose a training method that maps each prompt as a dependency graph, so a model that fails a parent constraint automatically loses credit for any dependent ones too, giving a finer-grained training signal than a single pass/fail score.
The finding echoes a pattern seen across AI benchmarks: once a system saturates an easy test, the test stops being informative, and the field needs harder ones. Whether the dependency-aware reward approach translates cleanly beyond the two models tested here — SD3.5-Medium and FLUX.1-dev — is the next question.