A new benchmark finds that even strong AI models often can't tell a technically sound code-review comment from one that just sounds right.
The paper, "CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?" (arXiv:2609.37216, posted September 30, 2026 as a preprint), introduces a 1,199-instance test set built from real pull requests, mixing genuinely correct review comments with plausible-but-wrong ones created through expert-verified perturbations. The researchers also built Sentinel, an agentic judge that pulls evidence from the repository before ruling on a comment, trained on Qwen3-Coder-30B-A3B-Instruct via iterative action-level learning from what the paper calls a "privileged teacher." On the benchmark's 359-instance test split, Sentinel scored 76.60% accuracy - 6.13 points ahead of general-purpose GLM-5.3 and 19.78 points ahead of its own untrained base model.
That gap matters because AI-generated review comments are already showing up in real pull requests, and a wrong-but-confident one can send a developer chasing a bug that isn't there, or waving through one that is. The paper's headline point - that general-purpose models struggle here even though they write fluent reviews - is a useful reminder that sounding right and being right are different skills for a language model.
Even Sentinel's purpose-built, repository-grounded 76.60% leaves it wrong roughly one time in four, so treat any AI review comment, however confident, as a suggestion to verify - not a verdict.