RAG pipelines look healthy until you check what's happening upstream.
Researchers ran 56 controlled experiments on a retrieval-augmented generation system, using a fixed 500-question QA set mapped to nearly 21,000 unique corpus contexts. They varied chunk size, retrieval depth, embedding-based reranking, and injected probabilistic noise into retrieval. The results were messier than standard benchmarks suggest: broader retrieval settings improved retrieval-oriented metrics, but exact-match and F1 scores moved non-monotonically — meaning more retrieval didn't reliably produce better answers. Smaller chunk sizes caused the system to lose answers during preprocessing, before retrieval even ran.
The core finding is that RAG evaluations anchored to final answer accuracy are flying blind. Failures accumulate at distinct stages — preprocessing, retrieval, context packing, generation — and a single aggregate score masks which stage broke. The researchers also logged higher variance under broader retrieval regimes, which means repeated runs on the same input can produce inconsistent outputs.
Most production RAG deployments are still benchmarked the easy way: feed in a question, check the answer. This paper argues that's roughly equivalent to judging a car's engine health by whether it reached the destination.