A research framework called FalconTrack cuts the manual labor out of training drone perception systems — and still hits near-perfect tracking accuracy.
The system, detailed in a new preprint, combines an automated labeling pipeline with a physics-aware tracking module designed for GPS-denied environments. Instead of hand-annotating thousands of images, FalconTrack uses a Gaussian Splatting simulator to isolate target objects from short video clips, composite them onto randomized backgrounds, and automatically generate RGB, mask, class, and 6-DoF pose labels — roughly 10,000 images in under 20 minutes. That labeled dataset trains a multi-head perception model, whose outputs feed into an Extended Kalman Filter with class-conditioned dynamics priors for real-time tracking.
The sim-to-real gap has historically been the graveyard of simulation-trained robotics models — systems that ace benchmarks indoors then fail outdoors. FalconTrack reports 96-100% class accuracy on zero-shot sim-to-real transfer across three geometrically distinct objects and two environments, and 100% success in closed-loop hardware tracking at around 25 Hz. A mask-centered baseline dropped to 60% success under fast out-of-view conditions, which is the scenario that tends to matter most in real deployments.
The broader context here is a race to make drone autonomy work without GPS or heavy pre-annotation budgets — a problem that matters in contested airspace as much as in consumer delivery logistics. Whether FalconTrack's results hold across more object types and cluttered real-world scenes is the question the preprint leaves unanswered.