A new framework for synthesizing long-context training data outperforms the fine-tuned models it was built to improve.
Researchers propose LongCrafter (arXiv:2607.06160), a pipeline that generates supervised fine-tuning data for large language models handling long documents. The system organizes long-context understanding into a two-tier taxonomy of 32 task types - spanning shallow retrieval up through deep cross-paragraph reasoning - then uses that taxonomy to decompose source text into evidence graphs that map dependencies between passages. Instruction-response pairs are generated only from traceable spans within those graphs, keeping answers grounded and difficulty tunable. Models fine-tuned on LongCrafter data outperformed both standard SFT baselines and the vendors' own post-trained versions of Qwen2.5-7B and LLaMA-3.1-8B across LongBench, LongBench v2, and LooGLE.
The "lost in the middle" problem - where models reliably surface information at the start or end of a document but miss what sits between - is a persistent headache for anyone deploying LLMs on contracts, research papers, or large codebases. LongCrafter's evidence-graph structure directly attacks position bias by training models to locate evidence wherever it appears, not just near the edges.
The largest gains appeared on the hardest tasks, which is the right place to improve - though benchmarks and real-world documents have a long history of not agreeing with each other.