A new benchmark says AI models are decent at sketching agent workflows and bad at making them actually work.
Researchers built MCPGen, a set of 100 self-contained projects across 16 domains, to test large language models on Model Context Protocol workflows - the connective tissue that lets AI agents call external tools. They ran 11 models through three tasks: rebuilding a workflow, creating a new tool, and extending an existing workflow without breaking it. Models hit 88.5 percent success on reconstructing workflow structure. When researchers checked whether the generated code actually ran, end-to-end execution success topped out at 57 percent for the best model. Individual tool tests passed 63.8 percent of the time, but full project integration succeeded less than 45 percent of the time.
That gap is the real story. It is one thing for a model to draw a plausible-looking pipeline; it is another for the tool implementations, schema bindings, and runtime wiring to actually agree with each other. This matters right now because MCP has become the default plumbing for AI agents that need to touch databases, APIs, and file systems - and vendors keep shipping agent features as if the wiring problem is solved. MCPGen's numbers say it mostly is not.
Unit tests passing while integration tests fail is a familiar pattern to any engineer who has shipped microservices - the components work alone and then argue with each other in production. AI agents are now running into the same wall.