Researchers have a new way to decide which medical exam answers deserve a written rationale, and the payoff shows up in consistency, not raw accuracy.
The method, called RMS-RSP, works by nudging a model's hidden states at the rationale tokens and checking how much that shifts the confidence gap between the correct answer and the best wrong one. Bigger shifts flag questions where an explanation is likely to matter most, useful when rationales are expensive to write or validate and only a limited token budget exists for training on them. The researchers tested it on five medical QA datasets using MedGemma-4B-IT across three training seeds, comparing it to ten other budgeted selection methods and a reference model trained on every available rationale. Under a fixed budget, RMS-RSP hit 60.61% accuracy on average versus 60.08% for random selection, a gain that only cleared statistical significance on one dataset, AfriMed-QA.
The real signal shows up when questions get scrambled. After reordering answer options three times, RMS-RSP improved robust accuracy by 1.91 points and semantic consistency by 2.85 points on average, in the same direction across all five datasets. That is a narrower but more useful claim than simply smarter training data: selectively chosen rationales make models less likely to flip their answer when a question is phrased differently, without needing the full, token-hungry rationale set.
Training on every rationale in the pool did push accuracy higher, to 63.74%, but at 29 to 254 times the token cost and without a consistent robustness payoff, a reminder that throwing more supervision at a model is not the same as making it reliable.