A new open-source model squeezes fresh tricks into the humble transformer block - and claims no compromise at inference time.
Wiola is a 13-million-parameter, decoder-only language model built for the 10-to-100-million-parameter range used in on-device inference and fast experimentation. Instead of just shrinking a standard transformer, its authors rework three parts of every layer. Spiral Rotary Positional Encoding tweaks the usual rotary frequencies so phase trajectories fan outward, aiding long-range discrimination without adding parameters. Gated Spiral Attention adds a per-head, content-adaptive gate built from a running statistic of the query stream, acting as a cheap, differentiable form of head selection. A Butterfly feed forward block swaps the standard expansion layer for a multiplicative interaction with an internal bypass, matching a four-times gated linear unit in parameter count while helping gradients flow through shallow stacks.
Most small language models are just scaled-down copies of architectures built for billion-parameter models, tuned for a regime they were never designed for. Wiola's authors instead adapt the math to the small-scale case, and they back it with a proof that training and cached autoregressive decoding behave identically, so nothing is approximated once the model ships. That kind of guarantee matters more as small models move from research demos into real products.
The catch: everything so far has been validated on a standard tiny-story corpus, which is a long way from proving these tricks hold up on messier, real-world small-model workloads.