A team successfully checkpointed a 7B model fine-tune across two free GPU sessions, then discovered their fine-tuned result was less accurate than the model they started with.
Using QLoRA on Mistral-7B-Instruct, the researchers split three training epochs across a Kaggle Tesla P100 session and a Colab T4 session by transferring only the 41.9M-parameter LoRA adapter between machines. Optimizer and scheduler state did not need to move. The cross-GPU handoff worked cleanly. The evaluation did not. A blind LLM-as-judge preferred the un-fine-tuned base model on 46% of test prompts versus 18% for the fine-tuned version. A factuality audit found four confident errors on policy-sensitive topics from the fine-tuned model; the base model produced zero. BERTScore did improve by 0.063, but that measured closeness to the training distribution, not correctness.
The culprit was the training data itself. The researchers generated their synthetic advising dataset with Gemini, then audited it using the same factuality method applied to the model outputs. They found verifiable errors in 28-40% of randomly sampled responses. Every confident mistake the fine-tuned model made traced back to an error already present in its training answers, not to anything introduced by the adapter-handoff technique.
Synthetic data generated by large language models has become the standard shortcut for cheap, domain-specific fine-tuning. This paper puts a number on the risk: nearly a third of responses wrong in a random audit, with those errors faithfully reproduced downstream. The BERTScore went up. The model got worse. Goodhart's Law, in writing.