A new training trick lets researchers fine-tune large language models without backpropagating through most of the network - and it roughly triples throughput while using less memory.
The method, called Forward-Pass-Only MLP training (FPO), starts from an observation: in the late layers of a transformer, the error at the model's output layer already approximates the true gradient. Across six public models, the two turned out to be correlated with a cosine similarity of 0.47 to 0.59 - close enough to be useful, not close enough to be exact. The researchers built a two-minute diagnostic that checks, layer by layer, where this approximation holds. FPO then takes a single error signal from the output and applies it directly to those layers, skipping the backward pass and the computational graph that normal backpropagation needs. Tested on OLMo-2-7B, Qwen3-8B, and Falcon3-7B, it delivered 2.7x to 3.2x the throughput of standard fine-tuning and cut peak memory by about 40 percent, while improving in-domain perplexity.
The more interesting number isn't the speedup - it's what stayed the same. On MMLU, ARC-Challenge, HellaSwag, and Winogrande, FPO-tuned models scored within seed-noise of the untouched baseline. Standard full-network fine-tuning, the paper notes, does not reliably manage that. For anyone building specialized versions of open models, that's the harder problem: making a model better at one thing without quietly making it worse at everything else.
Worth noting: this is one arXiv preprint, not yet peer-reviewed, and the underlying approximation is a correlation in the high 0.4s and 0.5s, not a substitute for the real gradient. A localized version of ordinary fine-tuning gets similar results but costs 2.2 times as much in wall-clock time - so the savings here come specifically from ditching the backward pass, not just training fewer layers.