The backbone comparisons dominating computer vision research may have been measuring the wrong thing all along.
Researchers introduced LUMA, a Lightweight Universal Mask Adapter, to solve a persistent confound in image segmentation benchmarks: every backbone under comparison ships with a different decoder, training recipe, and pretraining scheme, so reported accuracy gaps tell you little about the backbone itself. LUMA is a single, lightweight mask-transformer head that attaches unchanged to any backbone — isotropic, hierarchical, convolutional, or mixture-of-experts — and treats it as a black-box feature extractor. Holding that head fixed, the team benchmarked 20 backbones across 11 pretraining schemes and multiple resolutions on the ADE20K and Cityscapes datasets. LUMA matches the accuracy of EoMT, currently the state-of-the-art efficient ViT segmenter, at lower computational cost.
The findings cut against two popular narratives in the field. First, "efficient" token mixers — architectures pitched as faster alternatives to standard attention — fail to deliver efficiency gains at the high resolutions that are supposedly their motivation. Plain ViT holds the throughput Pareto-front at every resolution tested. Second, and more striking: pretraining objective, not architecture, is the dominant lever for segmentation quality — meaning years of architecture search may have been tuning the wrong knob.
This is the kind of result that makes a lot of conference papers look shakier in hindsight. If decoder choice was quietly driving benchmark wins, the architecture rankings that shaped funding decisions and follow-on research need a second look.