A new benchmark reveals that multilingual leaderboards are a poor guide for picking text embedding models in Portuguese.
Researchers released MTEB-PT, a Portuguese-language evaluation suite built from a subset of the broader MMTEB collection. It covers 14 datasets across four task types: semantic textual similarity, classification, retrieval, and reranking. The team ran 17 open- and closed-source embedding models through the same protocol and found that performance varied sharply by task — no single model won across the board, and a model's multilingual ranking was not a reliable predictor of how well it handled Portuguese specifically. Models with stronger long-context handling had a clear edge on retrieval and reranking tasks.
The findings matter because Portuguese has over 200 million native speakers, yet embedding model selection has historically defaulted to English or aggregate multilingual scores. That shortcut, the benchmark suggests, leads practitioners to pick the wrong tool. Language-specific fine-tuning still delivers measurable gains — particularly when the adaptation data matches the target task type.
To put numbers to that claim, the team fine-tuned three backbone models using Portuguese contrastive supervision and Matryoshka Representation Learning, a technique that lets embeddings be truncated to smaller dimensions without catastrophic quality loss. The biggest gains showed up on similarity tasks, which matched the training signal most closely. The benchmark, models, and code are all public — a rare case where the researchers leave you with something you can actually run.