Retrieval-augmented generation has a ranking problem that nobody talks about enough.
Researchers have introduced ReRanking Preference Optimization (RRPO), a reinforcement learning framework that retrains the reranker component of RAG pipelines using feedback from the downstream language model rather than static human relevance labels. Instead of asking "is this document topically related," RRPO asks "did this document help the LLM produce a better answer" — and optimizes from there. The system treats reranking as a sequential decision-making problem and uses LLM-generated quality signals as its reward, sidestepping the cost of human annotation. To keep training stable, the team added a reference-anchored deterministic baseline. Benchmarks show RRPO outperforming RankZephyr, a strong list-wise reranker, and the framework tested cleanly across different reader models including GPT-4o and alongside query expansion tools like Query2Doc.
The gap RRPO targets is real and underappreciated: most RAG systems bolt together an off-the-shelf retriever, a separately trained reranker, and a language model, each optimized in isolation. A document can rank highly on information retrieval metrics while being nearly useless to the LLM trying to synthesize a precise answer. Closing that loop with RL feedback rather than proxy labels is a sensible direction, and the fact that it transfers to multiple reader models suggests it is learning something general rather than overfitting to one LLM's quirks.
RAG pipelines have accumulated a long stack of independently trained components, and this is one more paper pointing out that the seams between them are where quality leaks out.