[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-gui-agents-trust-the-dom-more-than-their-eyes":10,"sections":34},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":29,"feedback":33,"feedback_at":22,"cost_usd":33,"total_tokens":33},3687,"gui-agents-trust-the-dom-more-than-their-eyes","GUI Agents Trust the DOM More Than Their Eyes","A new benchmark finds that multimodal GUI agents systematically ignore screenshot pixels in favor of accessibility-tree structure when the two conflict.","When a GUI agent looks at a screen, it often isn't really looking.\n\nResearchers have published a study diagnosing how multimodal GUI agents form beliefs about interface state. These agents receive two redundant signals: rendered pixels from a screenshot and a serialized structure like a DOM or accessibility tree. Across 310 probes spanning web, mobile, and desktop interfaces, every model tested deferred to the structural text channel when the two sources disagreed — even when the pixel channel was correct. The study introduces a metric called the Perception-Fusion Gap, which measures how often an agent perceives something accurately in the image yet still resolves toward the structural data. For textual conflicts, that gap is positive across all five models from three vendors.\n\nThe finding matters because it exposes a hidden assumption baked into how these agents are evaluated. Most existing benchmarks score task success or element grounding — they don't ask *which* input channel drove the decision. An agent can score well on standard tests while being functionally blind to the visual layer. The researchers found that in two live environments, the structural-over-pixel substitution produced wrong actions and real task failures, not just benchmark noise.\n\nThe effect is specific: it shows up in agents that use serialized-text and indexed-action interfaces but largely disappears in coordinate-action agents that must work directly with pixel positions. A white-box ablation traced textual conflicts to a single copied structural value — a narrow, auditable failure mode, which is either reassuring or alarming depending on how many production systems are quietly running textual agents right now.","[\"ai\",\"gui-agents\",\"benchmarks\",\"multimodal\"]","2026-07-07T04:00:00.000Z","2026-07-07T05:47:32.186Z","2026-07-07T05:47:35.142Z","published",null,[],"ai",[24,26,27,28],"gui-agents","benchmarks","multimodal",[30],{"name":31,"url":32},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.04334",0,{"sections":35},[36,40,45,50,55,60,65,70,75,79,84,88,93,98],{"name":37,"slug":24,"count":38,"latest_published_at":39},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":41,"slug":42,"count":43,"latest_published_at":44},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":46,"slug":47,"count":48,"latest_published_at":49},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":51,"slug":52,"count":53,"latest_published_at":54},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":56,"slug":57,"count":58,"latest_published_at":59},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":61,"slug":62,"count":63,"latest_published_at":64},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":74},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":76,"slug":77,"count":78,"latest_published_at":18},"Dev Tools","dev-tools",59,{"name":80,"slug":81,"count":82,"latest_published_at":83},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":85,"slug":86,"count":82,"latest_published_at":87},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":89,"slug":90,"count":91,"latest_published_at":92},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":94,"slug":95,"count":96,"latest_published_at":97},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":99,"slug":100,"count":101,"latest_published_at":102},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]