The go-to test for figuring out whether an image model thinks in shapes or textures turns out to have a measurement problem.
A team of researchers audited the cue-conflict benchmark — the widely used tool for probing whether neural networks lean on shape or texture when classifying images. They found three compounding flaws: the stylization technique used to separate visual cues does not reliably produce cues that are perceptually clean or equally informative; the ratio-based scoring can hide how sensitive a model actually is to each cue in absolute terms; and evaluating only a narrow set of preselected image classes distorts results by ignoring the full label space a model operates in. The upshot is that published bias scores may reflect measurement artifacts as much as genuine model behavior.
This matters because shape bias has been treated as a proxy for robustness — the working assumption being that models with stronger shape preference perform better in-domain and generalize more like humans do. If the benchmark measuring that preference is noisy, conclusions built on it are shaky, including design choices baked into training pipelines and architecture comparisons across the field.
The researchers introduce REFINED-BIAS, a dataset and evaluation framework built around explicit definitions of shape and texture, human- and model-verified cue pairs, and a ranking-based metric that spans the full label space. Whether it displaces the old benchmark depends on adoption, and the history of ML evaluation is littered with fixes that never quite caught on.