Researchers have built a faster way to strip layers from vision Transformers while keeping accuracy nearly intact.
Most compression work on vision Transformers targets width pruning — shrinking layers rather than removing them outright. Depth pruning, which cuts entire layers, offers bigger speed gains but has historically wrecked accuracy, so engineers avoided it. A team has now traced that failure to a overlooked cause: existing depth pruning methods ignore how much individual layers differ from one another, creating dimension mismatches when layers are removed. Their method, HetDPT, accounts for that heterogeneity explicitly. On ImageNet-1K, CIFAR-100, COCO, and ADE20K benchmarks, HetDPT delivers a 1.58x speedup for the DeiT-B model and 1.39x for DeiT-S with almost no accuracy drop.
The harder number is what happens at the extreme end. Paired with width pruning, HetDPT+ pushes the acceleration ratio for one standard configuration from 4.24x to 5.19x — a meaningful jump — while staying near-lossless on accuracy. That combination matters because most real deployments care about inference cost, not benchmark elegance, and prior joint pruning methods hit a wall at high compression ratios.
Depth pruning has been the unloved sibling of model compression for years precisely because accuracy recovery was so unreliable. If HetDPT's results hold outside controlled benchmarks, it could shift how practitioners think about slimming down vision models — though the gap between an arXiv result and a production deployment is where most promising methods quietly disappear.