A new benchmark asks a simple question: does that erased object in your video actually look gone, or just smudged?
Researchers released VOR-Bench, which bundles three pieces. VORD is a paired-video dataset with edited clips and masks drawn from model-generated, tool-rendered, and camera-captured sources. rMPAF is a framework that combines image-based object removal with fine-tuned video generation models to automatically produce realistic, motion-coherent paired videos. VOR-MDSM is a vision-language-model-based scorer trained across three evaluation dimensions to match human judgment rather than arithmetic pixel comparisons. The team reports its scores correlate with human ratings above 0.9, and plans to release the full benchmark and documentation for reproducibility.
That matters because current video object removal is often graded by metrics that check whether an edit matches a reference frame, not whether the erased area still tracks moving shadows, reflections, and occlusion as the camera moves - the details that actually convince a viewer nothing was ever there. Pairing a mixed-source dataset with a perception-aligned scorer gives developers a more honest yardstick than synthetic-only benchmarks typically offer.
That 0.9-plus correlation is the paper's own reported number. Whether VOR-MDSM holds up once outside researchers start scoring their own models against it is the actual test.