A new paper shows that throwing more context at large language models does not fix their evidence problem.
Researchers introduce Bactrainus, a modular selector-reader framework for multi-hop question answering, tested on the HotpotQA distractor set. Instead of dumping every candidate paragraph into one prompt, it splits the task into three stages: picking relevant paragraphs, identifying supporting sentences, and generating the final answer. The team fine-tuned Llama 3.1 8B and 70B Instruct models as readers, and tested optional question decomposition and teacher-generated rationale supervision on top. When the system was given the full candidate context instead of hand-picked supporting facts, answer F1 dropped 17-21 points even for the 70B model, while the strongest configuration overall hit 89.01 answer F1 and 79.70 joint F1.
The result undercuts the assumption that bigger models simply reason past noisy context, since scale did not close the gap between gold evidence and full context. What actually moved the needle was adapting the reader model itself and controlling evidence at the sentence level; fancier reasoning add-ons like decomposition and rationale supervision produced smaller, inconsistent gains.
It is a reminder that in the age of retrieval-augmented systems, curating what a model sees still beats trusting it to sort the signal from the noise.