A new transformer architecture ditches global attention for local blocks and a compressed memory trail.
Researchers have introduced BCMT (Blockwise Causal Memory Transformer), a language model architecture built to sidestep the quadratic cost of standard self-attention. Instead of comparing every token to every other token, BCMT runs dense attention only within small local blocks. Each block then produces a summary, and those summaries get folded into an exponential causal memory that flows forward into later blocks. In tests on context lengths up to 1024 tokens, BCMT matched the validation performance of standard dense transformers while training faster and using less memory, and an ablation study traced those gains specifically to the memory mechanism.
This targets a real bottleneck. Attention's quadratic scaling is the reason long-context models are expensive to train and run, and it's why linear attention, state-space models, and recurrent memory schemes keep showing up in the research literature. BCMT's pitch is compatibility: it slots into existing dense-attention implementations rather than replacing them outright, which matters more for adoption than the underlying math.
The catch is scale. 1024 tokens is a modest context window by 2026 standards, when production models routinely advertise context windows in the hundreds of thousands. Whether this memory trick holds up, and stays cheap, at that length is the test that actually matters.