Modern language encoders are worse than decade-old BERT at a core search task - and researchers now say they know why.
A new paper identifies what the authors call a "Vocabulary Gap" as the culprit behind a persistent anomaly: advanced encoders like ModernBERT, which beat BERT handily at dense retrieval, fall short of it on learned sparse retrieval. The problem lies in how modern tokenizers work. They use raw, case-sensitive vocabularies built for lossless text reconstruction, which means a single concept can map to several redundant surface forms. That wastes model capacity on noise that doesn't help lexical matching.
The finding matters because sparse retrieval is not a niche problem. It underpins keyword-style search across enterprise systems, legal databases, and biomedical corpora where exact term matching carries real weight. If newer encoders can't handle it well, practitioners are stuck running older, weaker models or patching together hybrid pipelines.
The researchers propose a fix called Vocabulary Transfer, a model-agnostic framework that swaps in a normalized, sparse-friendly vocabulary with minimal retraining cost. Two mechanisms do the heavy lifting: Semantic Initialization preserves geometric relationships between tokens during the swap, and Activation Potential Calibration keeps neurons from going dead or collapsing into dense outputs. Applied to ModernBERT, the approach hits 52.4 nDCG on the BEIR benchmark, a 4.7-point gain. It also revives RoBERTa-large, which the paper describes as a "failing" model on this task.
The key takeaway is architectural: the performance gap was never baked into the models themselves, just into a vocabulary design choice that prioritized reconstruction over retrieval. That's a solvable problem, though it's worth noting the code is currently hosted on an anonymous repository - peer review is still ahead.