An AI framework built for academic peer review wants to stop language models from making things up when authors write rebuttals.
The system, called RebuttalAgent, is the core contribution of a paper titled Paper2Rebuttal. Rather than feeding a reviewer's comments directly into a model and hoping for the best, RebuttalAgent decomposes complex feedback into individual concerns, then builds evidence before writing a single word. It pulls from compressed summaries and high-fidelity excerpts of the original manuscript, and can reach out to external literature when a reviewer's point requires outside support. A response plan is generated and inspected before any prose is drafted — meaning a human can see the reasoning chain, not just the output.
The distinction matters because hallucination is a known failure mode when LLMs handle dense, citation-heavy academic text. Existing tools tend to treat rebuttal writing as a text-completion task, which is exactly where models confidently fabricate supporting evidence. An evidence-first architecture at least makes the fabrication easier to catch.
The authors tested RebuttalAgent against a benchmark they also built — RebuttalBench — and report gains in coverage, faithfulness, and what they call strategic coherence. Self-reported benchmarks on self-built datasets are a peer-review tradition in their own right, so independent validation would be a more convincing next step.