Keeping recurrent neural networks from forgetting things turns out to be partly a geometry problem.
A newly published technical post argues that applying matrix orthogonalization to the weight matrices inside recurrent models substantially improves their ability to retain information over long sequences. The core claim is that orthogonal matrices preserve gradient magnitudes during backpropagation, which is where standard recurrent networks typically fall apart — gradients shrink toward zero as they travel backward through time, starving earlier layers of useful signal. The author walks through the math and benchmarks the approach against baseline recurrent architectures, finding measurable gains in sequence recall tasks.
This matters because the AI field has largely moved on from recurrent networks in favor of transformers, but recurrent and state-space models are staging a quiet comeback — largely because they handle long sequences at a fraction of the memory cost. Techniques that close the performance gap without adding architectural complexity are worth paying attention to, especially as researchers look for alternatives to attention-heavy designs.
Transformers solved the long-range dependency problem by brute-force attending to every token, but that approach scales quadratically with sequence length. If orthogonalized recurrent weights can recover some of that capability cheaply, the architecture might earn a second look in resource-constrained settings.