The rules used to align AI models are less precise than they look.
Researchers studying Constitutional AI methods found that compressing human preference data into short lists of natural-language principles leaves a critical gap: the principles say nothing about how to resolve conflicts between them. Using three benchmark datasets — PRISM, AlpacaEval, and Chatbot Arena — the team identified three concrete problems. Principle quality is hard to measure; existing proxies like coverage and accuracy don't predict real-world reconstruction. Composition is ambiguous: two different executors applying the same principles agreed only 73% of the time. And constitutions aren't portable — cross-model agreement sat at 73%, while the same model judging its own outputs reached 81%.
This matters because Constitutional AI and its variants underpin alignment work at several major labs, and the assumption has long been that a well-written list of principles is close enough to a decision rule. It isn't. A 7-point gap between intra-model and cross-model agreement means that what counts as "aligned" shifts depending on which model is doing the judging — a problem that scales badly as labs deploy these systems broadly.
The paper's proposed fix, a refinement step called ICAI+, nudges inter-executor agreement from 73% to 78% and brings transparent executors within a point of LLM judge accuracy. That's progress, but it's incremental — and the authors frame their work explicitly as open problems, not solutions.