AI health chatbots were built for a patient who doesn't exist.
Researchers analyzed 2,053 real conversations between patients and health chatbots powered by large language models, then compared them against the tidy, cooperative exchanges most systems are trained and tested on. The gap was significant. Real patients express emotions unpredictably, communicate indirectly, and don't always describe symptoms in clinical terms. To measure the problem, the team built a patient simulator that models clinical content, emotional state, conversational strategy, and communication style separately. In a Turing-inspired realism test, 15 human graders could only tell simulated conversations from real ones 55% of the time — barely above a coin flip. Across 1,164 clinician-graded cases using five distinct patient personae, the researchers evaluated four LLMs on urgency assessment and found that communication style alone could meaningfully shift triage outcomes.
That last finding is the one that should worry health systems deploying these tools. If the same underlying symptoms yield different urgency ratings depending on how a patient phrases them, the chatbot isn't measuring clinical risk — it's measuring articulateness. Patients who are elderly, non-native speakers, anxious, or simply less comfortable with formal language would be systematically disadvantaged.
This isn't an edge case; it's the rule. Most people seeking medical help aren't calm, precise, or cooperative on demand. Building symptom-assessment AI on idealized interactions and calling it patient-centered is, at best, a marketing claim.