A spiking neural network just got close enough to GPT-2 to make the comparison interesting, and awkward.
Researchers built SymbolicLight V1, a 194M-parameter language model that fires sparse binary spikes instead of running dense matrix math on every token. It pairs a spiking Leaky Integrate-and-Fire mechanism with a continuous "residual stream" and a windowed attention mixer, trained from scratch on a 3-billion-token Chinese-English corpus. The result: perplexity around 8.9, more than 89% activation sparsity, and performance within shouting distance of a GPT-2 model trained on identical data and hardware, which scored 8.27. On five zero-shot benchmarks, the two are statistically indistinguishable.
Here is the part the abstract buries: on a measured RTX 2080 Ti, SymbolicLight used 2,848 millijoules per token versus 905 for GPT-2. That is over three times the energy, from a model built explicitly around the promise of spiking efficiency. Sparsity was supposed to translate into cheaper inference; on real hardware, none of the usual GPU kernels are built to exploit binary spikes, so the theoretical efficiency gets eaten by overhead.
That gap is the real story. Spiking language models have chased dense-Transformer quality for years and mostly fallen short; closing the perplexity gap to 7.7% is a genuine step. But it also confirms that "biologically inspired" and "energy efficient" are not the same claim, at least not until hardware exists that actually rewards sparse spikes instead of just tolerating them.