A 1.5 billion parameter model built for African languages just beat systems from Google, Meta and Alibaba that are up to eight times its size.
Vambo AI released MORENA, a 1.5B model covering 12 African languages plus English, French and code. Rather than fine-tuning an existing Llama or Gemma checkpoint, the team trained it from scratch with a custom tokenizer built around African languages instead of one designed for English and programming text. Vambo says MORENA scored 1.408 bits-per-byte on a key benchmark, the best of 26 models tested, edging out its closest rival's 1.423 despite that model carrying more than five times as many parameters. The company also released smaller 0.5B and 0.2B versions, the latter aimed at tasks like keyboard input and speech-recognition rescoring.
Most African-language AI projects bolt extra training onto an existing Llama or Gemma checkpoint, inheriting a vocabulary built for English and code that wastes tokens on African text. MORENA's custom tokenizer needs up to 1.53 times fewer tokens than Llama 3.2 for the same passages, which means cheaper training and inference without depending on a foreign base model's assumptions in the first place. That design choice - not just the benchmark scores - is the more interesting bet here.
The whole project reportedly ran to about 22,000 GPU hours and $40,000 in compute, with support from UNDP, AIHub4SD and CINECA - a reminder that closing Big Tech's language gaps doesn't always require Big Tech's budget.