Small AI models are getting much better at finding the right rule in a compliance rulebook. Proving they understood it is another matter.
Researchers built a three-stage retriever on top of LegalBERT, combining entailment tuning, contrastive learning, and a BM25 score fusion, then tested it on ObliQA, a question-answering benchmark drawn from Abu Dhabi Global Market rulebooks. The retriever pushed Recall@10 from 0.256 to 0.774, beating both plain BM25 (0.678) and the dense encoder E5-large-v2 (0.758). Paired with compact generator models, ranging from 2B to 12B parameters and squeezed down to 4-bit precision, the team applied retrieval-aware fine-tuning through LoRA, which lifted the composite RePASs quality score for every model tested, with the biggest jump on the weakest one.
Here is the problem. A closed-book version of the same generator, given no retrieved passages at all, scored within 0.011 RePASs of the full pipeline, despite citing nothing and misstating obligations outright. That gap is narrow enough to suggest RePASs rewards confident-sounding answers more than grounded ones. The adapted models also failed to transfer to an unrelated domain, Australian case law, meaning this fix is tuned to one rulebook rather than compliance reasoning in general.
For a use case where a wrong answer can mean a real regulatory violation, a scoring system that cannot tell grounded answers from confident guesses is the actual headline here, not the recall numbers.