Turns out teaching an AI agent to think fast isn't the same as teaching it to work well with others.
Researchers built a benchmark called ICRL4AHT to test whether "in-context reinforcement learning" - a technique that lets AI agents adapt to new situations without retraining - actually works for ad-hoc teamwork, meaning coordination with partners the agent has never seen before. The benchmark runs on a fast JAX-based version of the cooking game Overcooked-V2 and includes a wide roster of teammate types, both reinforcement-learning-trained and rule-based. The team tested two leading approaches, Algorithm Distillation and Decision-Pretrained Transformer, across millions of recorded interactions. Both frequently did worse than a random baseline when facing new teammates or new kitchen layouts, and neither improved meaningfully the longer an episode ran.
This matters because in-context learning has been sold as a shortcut to flexible AI agents - no retraining needed, just show the model examples and let it adapt. That story has held up reasonably well for single-agent tasks. It falls apart the moment another unpredictable agent, human or otherwise, enters the picture, which is exactly the scenario behind every pitch for AI copilots, warehouse robots, and multi-agent assistants that are supposed to slot into existing teams.
If an agent can't reliably infer what a stranger is doing in a simplified cooking game, it's worth asking how much confidence to place in bigger claims about agents that "coordinate" with people in messier, real-world settings.