A research benchmark called HEARTS finds that large language models perform poorly on health time series data — and that scaling up models does not fix the problem.
The HEARTS benchmark, introduced in a paper posted to arXiv, pulls together 16 real-world datasets spanning 12 health domains and 20 signal types, such as EEG, ECG, and other physiological measurements. Researchers defined 110 tasks across four categories — Perception, Inference, Generation, and Deduction — and ran 16 state-of-the-art LLMs through more than 20,000 test samples. The verdict: specialized models built for time series analysis beat general-purpose LLMs by a meaningful margin. Worse, a model's score on standard reasoning benchmarks barely predicted how it would do here.
That second finding is the one worth sitting with. The AI industry has leaned heavily on general reasoning scores as a proxy for capability across domains. HEARTS suggests that proxy breaks down when the data is physiological and time-dependent — models fall back on simple pattern shortcuts and lose coherence as temporal complexity increases. Models from the same family tend to fail in the same ways, which implies the problem is architectural, not just a matter of training data volume.
The benchmark is designed as a "living" testbed, meaning researchers can keep adding tasks as the field moves. That is useful, because the gap it exposes — between marketing-ready general intelligence and the domain-specific rigor clinical applications require — is the kind that tends to get papered over until someone builds a test that makes it impossible to ignore.