A new academic benchmark called TSHA reveals just how poorly today's vision-language models perform when asked to identify safety hazards in real homes.
Researchers built TSHA — short for Trustworthy Safety Hazard Assessment — to fix what they say are three core failures in existing benchmarks: too much reliance on synthetic, simulation-generated imagery; hazard categories too narrow to reflect real-world variety; and no rigorous evaluation protocol for complex indoor scenes. The dataset includes 66,668 validated question-answer pairs pulled from real internet images, newly captured photos, AI-generated imagery, and Sora-generated videos, plus Hunyuan panoramic images that pack multiple hazards into a single scene. Testing across 22 popular VLMs, the researchers found current models consistently fall short on safety assessment tasks.
The gap matters because VLMs are increasingly pitched for home monitoring, elder care, and insurance inspection use cases — all high-stakes applications where a missed hazard has real consequences. Models fine-tuned on TSHA's training set improved by up to 18.3 points on the test set and showed better generalization on other benchmarks, suggesting the domain gap is fixable but has simply not been a priority.
The synthetic-data problem is a recurring theme in applied AI: models trained on tidy, computer-generated scenes often fall apart when facing the genuine chaos of a lived-in kitchen or cluttered hallway. TSHA doesn't claim to solve the underlying capability deficit — it just makes the deficit harder to paper over with a convenient benchmark.