A research team has released MECoBench, a benchmark designed to measure how well multimodal AI agents cooperate inside physically grounded environments.
The benchmark spans diverse real-world tasks and tests agents across two cooperation structures and three collaboration modes. Researchers ran extensive experiments across multiple multimodal large language models and pulled out three consistent findings: collaboration generally improves task completion, but the payoff shrinks when coordination complexity rises. Communication between agents drives most of the gain, and the best collaboration mode shifts depending on team size and model capability. The team also found that working together makes agents more robust when starting information is noisy or exploration conditions are uncertain.
Most MLLM benchmarks measure a single model working alone. MECoBench shifts the question to what happens when several multimodal models have to coordinate — a setup much closer to how AI is actually being deployed in robotics and autonomous systems. The finding that coordination overhead can erase collaboration gains is the kind of nuance that solo-agent evals simply cannot surface.
The code and dataset are public on GitHub, which is the right call — benchmarks only matter if rivals can replicate and poke holes in them.