An AI model can now estimate how big people and scenes actually are in a video, and keep track of who's who, without the slow scene-by-scene optimization older methods need.
The system, called WildHSR, builds on 3D foundation models that already reconstruct cameras and rough geometry from video in a single forward pass but have no sense of true scale, so a person in the output could look life-sized or shrunk to a foot tall. The researchers trained a small add-on, a Scale Readout head, to estimate metric scale from the foundation model's internal features, bootstrapping it on ordinary web video before fine-tuning on clips with exact scale measurements. Separately, they found the foundation model's existing internal features already encode which person is which across frames, so a tiny added projection can track individual people through a clip using that signal along with body motion and confidence scores. On the EMDB-2 benchmark, WildHSR became the first feed-forward method in the comparison to beat the best optimization-based approach on two accuracy measures, and on a second benchmark, RICH, it topped every other feed-forward method tested.
Most systems that reconstruct people and scenes together still rely on slow, per-video optimization to line things up. WildHSR does the whole job in one forward pass, running at 10.1 frames per second on a single GPU, which is the kind of speed that could make this sort of reconstruction useful for sports analytics or AR previews rather than just offline research pipelines.
It's a preprint, not a shipped product, and 10.1 fps is fast for this field but still well short of real-time, so the real test is whether the tracking holds up on video messier than the curated benchmark clips it was scored on.