New research argues most LLM safety screening can run on commodity CPUs, no GPU required for the bulk of production traffic.
Researchers evaluated five CPU classifier families — support vector machines and gradient-boosted trees trained on TF-IDF features — against GPU-based models including fine-tuned DeBERTa-v3 and Gemma-2B with LoRA, across three test regimes. On standard, in-distribution jailbreak prompts, the best CPU classifier matched the best GPU transformer at roughly one-fifth the deployment cost. Out-of-distribution prompts exposed a sharp failure mode: CPU classifiers produced high-confidence false negatives, letting novel attacks slip through without triggering escalation. Adversarially obfuscated inputs flipped the result — CPU models outperformed GPU transformers by more than 26 percentage points in F1. From those complementary failure patterns, the team built GuardChain, a three-stage pipeline that routes each prompt through regex, then a CPU classifier, then a GPU model, stopping at whichever stage can make a confident call.
The routing design is the paper's real contribution. Tiered inference — run cheap models first, escalate only what they can't handle — is standard practice in search and recommendation systems but largely absent from LLM safety pipelines. That the CPU and GPU stages fail in different directions makes them genuinely complementary rather than just redundant, which is a better justification for a hybrid system than cost alone.
The out-of-distribution failure mode the paper documents is precisely where a confident wrong answer is most dangerous, so getting confidence calibration right in production is real engineering work, not a footnote. Still, for teams currently routing all traffic through GPU-based safety filters, the implication is uncomfortable: most of that capacity is handling prompts a linear model could catch for a fraction of the price.