A new way of encoding information in spiking neural networks is closing some of the performance gap with standard language models.
Researchers introduce Spora, a spiking language model that changes how individual spikes carry information. Instead of just counting spikes over a time window, which caps information at roughly log2(T) bits for T time steps, Spora uses binary temporal weights so T spikes can carry up to T bits. Two encoding schemes make this work: Unipolar Binary Spiking (UBS), which uses thresholds and spike-triggered residual decay to produce non-negative integer codes, and Bipolar Binary Spiking (BBS), which separates sign from magnitude to handle signed values. Both feed into cheap accumulation-and-shift math instead of full multiplication inside the attention layers, the part of language models that is usually the most expensive to compute. Using just four time steps, Spora scores 76.6 on the GLUE benchmark suite and 44.1 on the CoLA grammar test, beating the prior SpikeLM model by 1.2 and 6.2 points. Stretching BBS out to six time steps pushes those scores to 78.2 and 47.4.
Spiking neural networks fire in discrete pulses rather than continuous values, which is why chipmakers like them for low-power hardware, but that same sparsity has made it hard to represent the nuanced features language tasks need. Spora's contribution is a more honest accounting of how much information a spike can actually hold, and it turns that extra capacity into math that stays cheap.
A 44.1 CoLA score is still a long way from what conventional transformer language models routinely post, so this is a win inside a specialized, power-constrained corner of AI, not evidence that spiking networks are about to replace anything mainstream.