Researchers have built a data-labeling pipeline for AI training sets that keeps GPU workers busy under uneven workloads and barely notices when machines crash mid-run.
The pipeline targets a problem anyone building large training sets now faces: labeling millions of text items by hand is not realistic, so teams lean on LLM "teacher" models instead, and then have to keep a fleet of GPUs fed without losing work when jobs fail. The team's scheduler, a work-stealing "ring pool", has each worker drain its own queue first, then pull tasks from its neighbors. Task claims use atomic compare-and-set writes on a single SQLite file, so failed jobs get detected and reclaimed automatically. A separate memory-aware rule caps how many model copies run per GPU based on available memory, so the same code scales across different hardware without retuning. They tested quality and cost with a relabeling benchmark: have the teacher model relabel a public dataset that already has gold labels, turning quality into a simple agreement score and cost into measured throughput.
LLM-as-teacher labeling has quietly become the default way many teams build training data, but most published work is about which model to use, not the plumbing that keeps GPUs fed while it does the labeling. This paper's numbers make the infrastructure case directly: under skewed workloads the pool hit up to 3.4 times the throughput of static sharding, and when half the workers were killed mid-run it lost zero of 2,000 tasks versus 953 lost by static sharding.
The whole system runs on a single SQLite file on one machine, which is what makes it dependency-free and easy to reproduce. It is still fair to ask whether that same trick holds up once you are coordinating hundreds of nodes on a real cluster, rather than simulating worker deaths on a laptop.