OWASP's ranked list of the biggest risks to LLM applications barely matches what actually happens in the wild - and that's not necessarily bad news.
Two members of the OWASP working group built a corpus of 7,714 snapshotted LLM-security incidents pulled from CVE, GHSA, OSV, and the AIAAIC database, then labeled 6,639 of them against OWASP's 20-entry risk taxonomy. Using a Bayesian model that corrects each category's count for how often a classifier over- or under-flags it, they built a separate incident-based ranking and compared it to the expert consensus behind the official Top 10. The two rankings barely agree: Cohen's kappa lands around 0.20, and the 90% interval crosses zero, meaning the true agreement could be close to nothing. The 2026 candidate list still leans on expert judgment, blending it with incident data at a fixed 75/25 split.
That mismatch sounds like a problem for the Top 10's credibility, but the stress test says otherwise. A pre-registered bake-off of four frontier classifiers found none could beat the incidence-based ranking's baseline balanced accuracy of 0.863, and that same ranking's order tracked held-out ground truth with a Spearman correlation of 0.918. Expert intuition about which LLM risks matter most is holding its own against a large real-world incident dataset, even where the two rankings disagree on specifics.
Worth remembering: this is a side analysis from two working-group members, not an official OWASP release, so don't expect the published Top 10 to get reshuffled because of it. But it's a useful reminder that risk lists built by committee aren't automatically out of step with reality - sometimes the incident data just hasn't caught up yet.