Researchers have a new answer to why AI harm detectors keep making inconsistent calls: they're guessing, not looking anything up.
A team published RoTRAG, a framework that retrieves human-written "Rules of Thumb" — short moral norms pulled from an external corpus — before each judgment in a multi-turn conversation. Instead of letting a language model rely solely on its training to decide whether something is harmful, the system fetches relevant normative principles and uses them as explicit evidence. A lightweight binary classifier decides whether a fresh retrieval is needed for each new turn or whether the existing context suffices, cutting redundant computation. On two multi-turn safety benchmarks, the approach produced around a 40% relative improvement in F1 score and an 8.4% reduction in distributional error over competitive baselines.
Most production content-moderation systems using LLMs suffer from exactly the problem RoTRAG is trying to fix: a model trained on internet data makes moral judgments based on statistical patterns, with no auditable trail of which principles it applied. Grounding those decisions in an external, inspectable corpus of norms doesn't just improve accuracy — it makes failures easier to diagnose and fix without retraining the underlying model. The question no paper answers is whose rules of thumb end up in that corpus, and how the system handles genuine cultural disagreement about what counts as harm.
The routing efficiency trick is the part worth watching — turning what would otherwise be an expensive per-turn retrieval into a selective one is the kind of practical engineering that moves safety research from benchmark paper to deployed system.