A new fine-tuning technique claims to beat LoRA's accuracy with a few hundred times fewer trainable parameters.
Researchers built a technique called LoRA-CRAFT (CRAFT for short) that fine-tunes large language models by freezing almost everything and training only a tiny slice of it. Most fine-tuning methods like LoRA add small trainable matrices on top of a model's weights and update those. CRAFT instead breaks the pre-trained attention weights themselves into compressed pieces, using a matrix trick called Tucker decomposition, then locks those pieces in place and trains only small square adjustment matrices around them. In tests on RoBERTa models and on LLaMA2-7B and LLaMA3-8B, CRAFT matched or beat standard LoRA's accuracy while training a small fraction of the parameters.
That matters because fine-tuning cost scales with parameter count, and CRAFT's edge over LoRA widens as models get bigger, not smaller - on LLaMA3-8B it beat LoRA's average accuracy using hundreds of times fewer trainable parameters. That is a different bet than most efficiency research, which usually trades away some accuracy for savings; here the paper reports no such tax, at least on the benchmarks tested.
It is an incremental step past PEFT cousins like LoTR and SuperLoRA rather than a clean break, and the paper's own benchmarks - GLUE and a handful of commonsense reasoning sets - are the ones every PEFT method clears before anyone tries it on harder, messier tasks.