A research team has built a multimodal benchmark that rewrites itself automatically, aiming to solve the contamination problem that makes most AI leaderboards unreliable.
MMBench-Live is a continuously updated version of MMBench, a widely used test for vision-language models (VLMs). Instead of a fixed question set that models can effectively memorize through training data exposure, the system uses a multi-agent pipeline to generate fresh evaluation instances on demand. Each update produces new question-answer pairs, costs roughly $30, and takes one to two hours to complete. The current dataset holds 5,900 generated instances, and the team says answer correctness rates are high. The project code is public on GitHub.
Static benchmarks have a structural flaw: once a dataset is public, it can leak into training data, making scores meaningless as a measure of genuine capability. MMBench-Live addresses this by extracting visual patterns from the original benchmark to guide new data collection, which keeps results comparable across versions while reducing the memorization signals that contamination produces. That comparability matters — a benchmark that changes its grading scale with every update is useless for tracking progress.
The $30-per-update figure is the number worth watching. If it holds at scale, it undercuts the usual excuse that keeping benchmarks fresh is too expensive — and puts pressure on labs that have quietly benefited from static tests staying static.