Multi-agent AI systems keep breaking in production, and a new benchmark suggests the problem usually isn't the model.
Researchers built MAFBench, a standardized test suite, and ran nine multi-agent LLM frameworks through it using the same underlying model, changing only the architecture: orchestration, memory, planning interfaces, specialization, and communication topology. Orchestration overhead alone multiplied latency by more than 60x in some frameworks, and a stripped-down version of the same coordination approach showed that cost came from sloppy implementation, not the approach itself. Forcing agents to output rigid, schema-constrained plans cut accuracy by up to 32 points - and the errors traced back to formatting failures, not bad reasoning. Mismatched communication topology dropped coordination success from above 90% to below 30%, and memory design controlled how well agents recalled information regardless of context window size.
That's a useful correction for anyone shopping for a multi-agent framework by which foundation model it's built on. The bigger predictor of whether a system holds together is how it's wired: who talks to whom, how state gets passed around, and how rigidly agents are told to format their output. It's the same lesson distributed systems learned decades ago - the protocol matters as much as any single node.
Worth noting: none of the nine frameworks tested could cleanly update what an agent already knows, so even the best-wired setup still can't un-teach a bad memory.