AI vision models look convincing until you ask them to do real physics.
Researchers introduced ImagingBench, a benchmark covering 20 computational imaging tasks across five categories: ray and wave optics, image signal processing, inverse reconstruction, computational sensing, and calibration. They tested leading vision-language models — including Gemini, GPT, and Qwen — against specialized, task-specific baselines. The models were evaluated in three settings: fixed expert-guided reconstruction, planner-guided reconstruction, and forward simulation for consistency checking.
The results were consistent and unflattering for the AI side. Agentic models lost to specialized methods across the board, with the worst gaps appearing in computational sensing problems like lensless imaging, holography, and time-of-flight imaging. Adding a planning layer helped only modestly — and inconsistently — over a simple fixed prompt.
That last finding is the one worth sitting with. The models frequently produced outputs that looked right to the human eye, but scored poorly on reference-based fidelity metrics. In other words, they can fake the aesthetic of a correctly reconstructed image without understanding the underlying physics. This matters because computational imaging is not a niche academic exercise — it underpins medical imaging, autonomous vehicle sensors, and scientific instrumentation.
The benchmark is a useful check on a pattern that keeps recurring: AI systems that ace pattern-matching tasks on natural images get exposed the moment a problem requires physically grounded reasoning. That gap is not closing on its own, and ImagingBench now gives researchers a common testbed to track whether it ever does.