A new method called X-FEMR tries to crack open the black box at the center of AI systems trained on patient health records.
Foundation models for electronic health records — pretrained on large-scale structured patient data — can convert longitudinal patient histories into predictions across a range of clinical tasks. The problem is that nobody can easily see why they reach a given conclusion. X-FEMR addresses this by training a separate Transformer-based surrogate model on input-output pairs from the original model, then identifying which individual tokens — discrete data points in the patient timeline — most influenced each prediction. The researchers also introduce a clinical alignment metric to measure whether the surrogate's key tokens match features that clinicians already recognize as meaningful.
Interpretability in clinical AI is not an academic footnote — it is a regulatory and safety issue. When a model flags a patient for sepsis risk or readmission, clinicians need a reason they can audit, not a confidence score they have to trust on faith. A token-level explanation that maps back to validated clinical features gives practitioners and regulators something concrete to scrutinize.
The surrogate approach is a well-worn technique in explainable AI — it is the same logic behind LIME and SHAP — but applying it specifically to health record foundation models with a domain-grounded evaluation metric is the novel step here. Whether the surrogate faithfully enough approximates the original model in edge cases, where clinical stakes are highest, is the question this paper raises without fully closing.