A new benchmark tests whether AI can watch a busy intersection like a human and correctly predict what happens next.
Researchers built RISE, a framework for tracking vehicles and pedestrians at intersections in 3D using ordinary calibrated cameras, no LiDAR required. The tracking method stitches together SAM3 video identities across multiple camera views using calibration-guided mask agreement, recovering persistent 3D tracks without retraining for each new intersection's layout. Tested on 20 human-reviewed clips across six intersections, it scored 66.9 MOTA, a standard multi-object tracking accuracy metric, within the study's defined evaluation scope. The team paired that tracking system with a new dataset, RISE-VQA: 33,910 question-and-answer pairs drawn from 557 clips across 16 intersections and 61 roadside camera views, built with a human-reviewed pipeline that grounds each question in bounding boxes without leaking future footage to the models being tested.
Camera-only 3D tracking matters because LiDAR remains expensive to deploy at scale, while calibrated cameras are already common at monitored intersections. If RISE's approach holds up beyond these test clips, that points toward metric-accurate tracking without new sensor hardware. But the paper's own companion benchmark, RISE-Bench, shows the harder problem is still unsolved: even with human-reviewed grounding, models struggled with spatial grounding, predicting future object positions, and reasoning about interactions between vehicles and pedestrians.
The tracking result is a solid engineering trick. The benchmark result is the more telling one: watching a scene and understanding what happens in it next are still different problems, and current vision-language models keep fumbling the second.