Text encoders have been stuck while the rest of AI moved on.
Since BERT arrived in 2018, encoder models have seen incremental updates but no fundamental redesign — even as decoder-based models scaled dramatically in size and capability. Researchers now argue that BERT's core architecture is the bottleneck. The problem: BERT couples its representation learning directly to a token reconstruction loss, which means the better the model gets at predicting masked words, the less useful its internal representations become for downstream tasks when you freeze the model and probe it. Better perplexity, worse embeddings — a quiet trap that has gone largely unaddressed.
The proposed fix is CrossBERT, a two-part architecture that breaks that coupling. One component handles high-quality representation learning; the other handles the token reconstruction task separately. The split unlocks two practical gains: a complementary masking strategy that lets the model train on masking ratios of 50 percent or higher, and gradient collection across all tokens rather than just the masked subset. The result is roughly 1.5 to 2 times better training throughput and twice the sample efficiency. On the MTEB English benchmark and frozen GLUE evaluations, CrossBERT shows monotonic scaling — meaning performance reliably improves as the model grows, which encoder architectures have historically failed to guarantee.
This matters because encoders do the quiet work that powers search, retrieval-augmented generation, and semantic similarity at scale. If CrossBERT's scaling behavior holds up in broader evaluation, it could shift how the industry builds the embedding models sitting behind most production AI pipelines today.
Decoder scaling got the headlines for years; encoders just got the diagnosis.