A new spiking neural network just learned language from scratch, no shortcuts needed.
Researchers built NeuronSpark, a self-attention-free language model made of spiking neurons whose decay rate, gain, and firing threshold adjust per token instead of staying fixed. That selectivity, paired with a gated associative memory and adaptive temporal aggregation, let the team pretrain a 1.1 billion parameter model on 2.1 billion tokens starting from random weights, with no conversion from an existing transformer and no distillation. On an eight-task zero-shot benchmark it scored 40.0, beating comparable-budget Pythia and Mamba checkpoints and holding onto 95.2% and 93.2% of those models' fully-trained performance respectively. The network also split itself into fast and slow processing tiers without being told to.
Most spiking language models to date have been converted or distilled from existing transformers rather than trained from zero, so a native pretraining recipe that actually competes is a real methodological step, not just a demo. The efficiency pitch, up to 90.1% lower energy-proxy use and far fewer compute cycles than Transformer or Mamba baselines at matched parameter counts, is the real selling point for anyone eyeing low-power inference.
But the number that matters for real use, throughput on the physical FPGA they built, comes in at 0.08244 tokens per second with a 24-second time to first token. A reminder that energy-efficiency papers and usable chatbots are still two different products.