Wide-angle fisheye cameras can now tap into state-of-the-art depth estimation without anyone retraining the underlying model.
Foundational monocular depth estimators are large models trained on tens of millions of standard perspective images — the kind a typical flat lens produces. Point one at a fisheye image, and the model struggles: the curved distortion shifts the statistical distribution of the input far enough from what the model expects that depth estimates fall apart. The new approach sidesteps that problem by introducing "Calibration Tokens," a small set of learnable parameters that adjust the model's internal representations so fisheye inputs look, to the model, more like the perspective images it already knows. No finetuning, no architectural changes.
The practical payoff is real. Fisheye lenses are common in robotics, automotive systems, and surveillance — anywhere a wide field of view matters — but they've been second-class citizens for depth perception because retraining large models for each camera configuration is expensive. A lightweight token-based adapter that generalizes across both indoor and outdoor scenes changes that calculus.
The method is self-supervised, meaning it doesn't need labeled fisheye data; it synthetically recalibrates perspective images to simulate fisheye distortion during training. That's a neat trick, though it also means the model's real-world ceiling is bounded by how faithfully that simulation captures actual fisheye optics — a question the paper leaves open.