Most medical AI question-answering systems can only read text — a problem when the answer is a diagram.
Researchers have introduced M3QAFrame, a framework designed to handle queries against medical documents that contain both written content and images. The system takes a question, its surrounding context, and associated images as input, then returns an answer assembled from relevant text passages and images together. It uses a transformer-based architecture to score the relevance of both sentences and images before assembling a response. The team also released a new dataset — M3QuestionIng — pairing medical queries with contexts, images, and extractive answers, each labeled by user intent and query type.
Most existing multi-span medical Q&A systems treat documents as plain text, which works until the clinically important content lives in a chart, scan, or figure. A system that can surface the right image alongside the right paragraph is meaningfully closer to how a clinician actually reads a reference document. The labeled intent and query-type metadata is also worth watching: it suggests the team is thinking about retrieval precision, not just raw accuracy.
The benchmark results show the approach outperforms prior methods, though "outperforms on evaluation metrics" is a claim that deserves scrutiny when those metrics were defined by the same team that built the dataset.