Shrinking a medical AI model to run faster can leave its final answer untouched while quietly gutting the reasoning behind it.
Researchers studied post-training quantization, the standard technique for compressing large language models so they need less memory and compute. Applied to medical question-answering models compressed down to 4-bit weights, activations, and key-value cache, a standard quantization method preserved the models' multiple-choice accuracy but substantially weakened the rationales those models gave for their answers. To fix that, the team built a method that also protects the specific tokens and reasoning steps supporting each answer, using a cache of the original full-precision model's explanations as a reference during compression. They tested it on four 7-8 billion parameter medical and instruction-tuned models across three medical QA benchmarks: MedExQA, MedExpQA, and ChallengeClinicalQA.
Accuracy has been the default yardstick for judging whether a compressed model is good enough, and this work shows that yardstick can hide a real problem. A model can land on the correct multiple-choice answer while its explanation no longer actually supports that answer, which matters most in domains like medicine where people are told to check the reasoning, not just the verdict, before trusting it.
It's a reminder that getting the right answer is a much lower bar than knowing why, and compression benchmarks have mostly been grading on the former.