A paper out of arXiv proposes that the activation function — a foundational building block of neural networks — is not as indispensable as textbooks suggest.
The researchers introduce Threshold Gating (TG), a primitive that routes computation through branches based on input-conditioned thresholds. They show that common activation functions — ReLU, GELU, SiLU, Sigmoid, Tanh, and others — are all special cases of this single TG primitive. Crucially, they converted pretrained CNNs, transformers, and recurrent networks to use TG without retraining and without performance loss. They also prove a "Minimal Branch Theorem" linking the minimum number of required branches to the trainability of deep networks.
The practical stakes go beyond theoretical tidiness. On the hardware side, TG maps cleanly to analog in-memory computing systems, sidestepping the analog-to-digital converter bottleneck that currently drives up power consumption and chip area — a real constraint as labs race to cut inference costs. The training-from-scratch results also hint at compression and speed gains that, if they hold at scale, would matter to anyone running large models.
The results are promising but the paper is a preprint, and "preserving model performance" on converted checkpoints is a lower bar than showing TG outperforms tuned baselines on frontier-scale architectures — that work is still ahead.