A tweak to LoRA fine-tuning squeezes out some of the accuracy that full fine-tuning still holds over it.
A new research paper proposes GDLoRA, a fine-tuning method that reworks how LoRA's low-rank adapters get updated. The authors show that LoRA's gradient updates are mathematically boxed in: they can only move in directions matching the adapter's current low-rank shape, leaving a chunk of the full weight gradient permanently out of reach. GDLoRA reconstructs that full gradient from the model's forward and backward passes, extracts the leftover piece, and applies it directly to the base weights, while the adapters keep training as usual with AdamW. Across tests on language understanding, math reasoning, commonsense reasoning, and image classification, GDLoRA beat standard LoRA and narrowed its remaining gap to full fine-tuning, without adding to the optimizer's memory footprint.
LoRA has spawned a cottage industry of variants, like DoRA, rsLoRA, and LoRA+, that all try to close the full fine-tuning gap by tweaking initialization or optimizer schedules, but they still operate inside the same constrained gradient space. GDLoRA's trick is different: it pulls in the information those methods throw away, without touching the memory savings that make LoRA appealing for teams without full-scale GPU budgets. That distinction matters if it holds up, since memory is usually the whole reason to pick LoRA over full fine-tuning in the first place.
The code link is anonymized, a signal this is still working through peer review, and the paper stops short of claiming parity with full fine-tuning; the gap is narrower, not closed.