A research framework wants to end the bottleneck between raw scientific data and usable AI training sets.
REDI is an open-source, five-stage pipeline — ingest, preprocess, transform, structure, and output — built for the kind of large-scale datasets that national computing facilities manage. Each stage logs provenance so results are reproducible, and the whole thing can be called as an agent-native skill. A companion tool, SetGo, handles FAIR compliance and catalog publication automatically. The team tested REDI across four domains: climate modeling, proteomics, materials science, and nuclear fusion, converting raw data to AI-ready outputs validated against domain-expert references.
The dirty secret of scientific AI is that most of the work happens before any model trains — cleaning, reformatting, and documenting datasets that were never designed for machine learning. REDI's preliminary benchmarks show near-ideal parallel scaling to 100 nodes on the Frontier supercomputer for the climate case, which matters because that kind of throughput is where manual pipelines fall apart. The provenance profiling also surfaced a practical finding: file I/O dominates pipeline cost, making format selection a more important variable than most practitioners treat it.
Frameworks promising to unify data preparation are not rare, but most stop short of agent-callable deployment or cross-domain validation. Whether REDI's coverage of four scientific fields translates to broader adoption — or stays a niche tool for HPC facilities — depends on how much of its setup burden it can actually absorb.