A research team has built a leaner way for AI agents to navigate graphical interfaces — skipping coordinate prediction entirely in favor of reading where the model is already looking.
GUI-AIMA is a fine-tuning framework that repurposes the attention maps already baked into multimodal large language models. Instead of training a model to spit out X/Y coordinates for a click target, the system identifies visually relevant screen patches first, then pins the action to the right spot within them. The team trained a 3-billion-parameter version on roughly 509,000 samples — about 101,000 screenshots — which is modest by the standards of this field. On five standard benchmarks, GUI-AIMA-3B hit 92.1% on ScreenSpot-v2 and 61.5% on the harder ScreenSpot-Pro, topping other models in its size class across the board.
Most GUI agents today treat screen interaction as a coordinate-generation problem, which demands both precision and large training sets. The attention-first approach matters because it exploits structure the model already has rather than bolting on a new skill from scratch — a meaningful efficiency gain if it holds up outside controlled benchmarks. A plug-and-play zoom stage can slot in without retraining, which lowers the cost of adapting the method to denser or smaller UI elements.
The 3B parameter ceiling keeps this accessible, but real-world GUI agents face interfaces that benchmarks rarely capture — dynamic content, overlapping elements, and localization quirks that static screenshots sidestep entirely.