Researchers have released a dataset that grades AI science agents on their reasoning, not just their final answers.
Called OpenDiscoveryTrace, the dataset logs complete step-by-step trajectories from AI agents tackling 124 scientific tasks in drug discovery, materials science, genomics, and literature analysis. Each step records nine fields: thoughts, tool calls, observations, errors, revision triggers, and confidence scores. The breakdown covers three frontier models (GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro) at 124 trajectories each, four open-weight models (Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, Qwen2.5-1.5B) at 30 each, plus 60 live-retrieval variants, for 552 trajectories total. It's released under CC BY 4.0 with a trace schema, agent harness, and five benchmark tasks with baseline models included.
Most AI benchmarks only check whether the output is right, which rewards lucky guesses as much as sound method. This one exposes the difference. In a pilot analysis of 363 trajectories, the three frontier models all succeeded 84-89% of the time, but Claude Opus 4.6 made 30 times more errors per trajectory than GPT-5.4 (2.5 vs. 0.08). Their mistakes looked nothing alike either: Claude's errors were mostly tool misuse, GPT-5.4's were mostly reasoning errors.
That's the real finding here: success rate alone would have told you these models are basically interchangeable for scientific work. It took the process trace to show they fail in opposite ways, which matters a lot if you're the one auditing an AI-generated hypothesis before it reaches a lab bench.