Searching GitHub for scientific software is harder than it sounds, and now there is a benchmark to prove it.
Researchers have released a curated corpus of 5,264 scientific repositories drawn from five NASA Science Mission Directorate divisions — Earth Science, Astrophysics, Planetary Science, Heliophysics, and Biological and Physical Sciences. On top of that corpus, they built two retrieval benchmarks: one with 219 expert-written queries for repository search, and one with 117,950 code snippets paired with 119,720 queries spanning seven programming languages. Both datasets are publicly available on HuggingFace.
Most code search benchmarks are built for general software engineering, which means they miss the domain-specific vocabulary scientists actually use. Baseline results here already expose the gap: performance varied substantially across scientific domains, driven by inconsistent documentation habits and different coding conventions between fields. That variation is the real finding — it quantifies a problem researchers have long complained about but could not measure.
General-purpose code search has attracted significant investment, yet the scientific community has largely been an afterthought. With over 600 million repositories on GitHub, the needle-in-a-haystack problem is acute for domain scientists who may lack the software engineering vocabulary to write queries that match how code is actually labeled. This benchmark gives tool builders a concrete target — though whether anyone with resources to act on it will prioritize scientific users over the larger commercial market remains an open question.