A new research paper cuts a small language model's parameter count by removing its output layer entirely.
Researchers behind RiLM (Riemannian Language Models) replace the standard output matrix - which can eat a third of a small model's parameters - with geometry. Instead of a separate weight matrix mapping hidden states to vocabulary scores, the model treats context as a path through a curved space and predicts the next token from the squared geodesic distance to each word's embedding. The same embedding table handles both input and output. Tested at roughly 290,000 parameters and a 2,000-word vocabulary, the hyperbolic version (HypRiLM) hit 54.2 validation perplexity on WikiText-2, beating a flat-geometry variant (87.6) and tied LSTM, Transformer, and state-space baselines that all landed between 113 and 147.
For a field obsessed with scaling up, this is a rare paper scaling down on purpose. Edge devices, offline assistants, and reproducible small-scale research all benefit from models that do more with fewer parameters, and cutting the output matrix is a structural fix rather than a pruning trick bolted on after training.
The catch: this is a controlled comparison on tiny vocabularies and toy datasets, not a benchmark against anything resembling a modern language model, so treat the 2x perplexity gain as a promising lab result, not a roadmap for your next chatbot.