A new open-source framework wants to fix one of the quieter problems in AI-assisted science: the research question that sounds rigorous but cannot be inspected.
FirstResearch wraps each proposed research question in what its authors call a Research Question Certificate — a structured document that records assumptions, a mechanism model, a falsifiable hypothesis, a minimal decisive test, and a failure update rule. The idea is that a scientist should be able to audit the question before any downstream experiment runs. The system was tested across ten LLM-agent research topics against prompt-level baselines drawn from three existing frameworks: AI co-scientist, Agent Laboratory, and AI Scientist-v2. Under a blind DeepSeek judge, FirstResearch outperformed all three. An independent Gemini-2.5-Flash rescore of the same 40 baseline packages confirmed the ranking: FirstResearch scored 4.86/5 versus 4.38/5 for the strongest competing baseline, with a Pearson correlation of 0.865 between the two judges on average score.
The more striking result is the ablation: stripping out the certificate entirely drops scores below 1/5 under both judges, while a certificate-only configuration scores 4.90/5 under DeepSeek and 4.88/5 under Gemini. That suggests the structured derivation record — not prompt engineering or model choice — is doing most of the work. Most AI science tools compete on ideation volume or literature coverage; FirstResearch is betting that auditability is the bottleneck researchers actually care about.
The authors are candid that these results are preliminary and rely on LLM judges rather than human domain experts, which limits how much weight any single benchmark score can carry — a caveat worth holding onto as the AI-for-science space gets louder.