Researchers just asked AI models to judge Olympic diving, and mostly, they flopped.
Researchers tested several open-source vision-language models in a zero-shot setup against the AQA-7 diving benchmark, asking them to reason about form and generate phase-level sub-scores. On their own, the models' scores correlated weakly with real judges' scores, with Spearman correlations below 0.32. The team then built a framework combining the models' text explanations, via TF-IDF vectorization and dimensionality reduction, with an ensemble regression layer. Stacking four models this way pushed the correlation up to 0.67, and notably, the models' written reasoning proved more useful than their raw numeric sub-scores.
This isn't AI replacing judges - it's proof that off-the-shelf vision-language models are mediocre judges alone but decent assistants when their reasoning is treated as a feature rather than a verdict. It also suggests that letting models explain themselves in text, then quantifying that text statistically, can beat trusting their final numeric scores directly - a lesson that likely applies well beyond diving boards, to any domain where AI renders a judgment call.
A 0.67 correlation beats guessing, but it's still far short of the consistency a panel of human judges delivers, so don't expect algorithmic scorecards at the next Olympics anytime soon.