A new academic paper proposes a leaner way to fine-tune large language models, sharing weights across layers like a sandwich instead of training every parameter from scratch.
Researchers introduce SAPE (Sandwich Adapters for Parameter Efficiency), a parameter-efficient fine-tuning framework that shares adapter weights across a model's middle layers while keeping the input embedding and final output layers separate and untouched. The design sidesteps two problems seen in earlier parameter-sharing approaches: uniform sharing across every layer, which slows convergence, and dynamic masking, which adds compute overhead. In tests on RoBERTa-large, SAPE outperformed a method called proPETL while using only 10% of its parameter budget. On LLaMA-3.2 3B under a roughly 0.6 million parameter cap, it beat AdaLoRA by 4.85 percentage points on the GSM8K math benchmark and 3.11 points on CommonsenseQA.
Fine-tuning costs remain a real bottleneck for anyone without a data center's worth of GPUs, so shaving trainable parameters without losing accuracy translates directly into lower memory and compute bills. The results also point toward a broader lesson: sharing weights based on a model's actual layer structure may matter more than clever masking tricks that add their own overhead.
The paper's own ablations flag a catch - locking shared weights in place helps general-purpose reasoning but slightly dulls the precise, step-by-step arithmetic multi-step math problems need, a reminder that parameter-efficient and best-at-everything are not the same claim.