Most servers handling AI inference or post-quantum crypto are wasting time waiting for hardware — and a new paper argues the fix is already sitting in the codebase.
Researchers tested ten production-grade servers spanning every major concurrency model and found that overlapping hardware offloads with other in-flight requests requires adding 22 to 138 lines of code and modifying at most one existing line. The approach skips the usual remedies — async framework rewrites, new runtimes, custom dataplanes — and instead treats the problem as routing: submit the offload, suspend the request using the server's own deferred-response primitive, then resume on completion. On real hardware, that change recovered 1.2x to 5.4x throughput. A more aggressive variant using an LD_PRELOAD fiber runtime, which injects the reroute into an unmodified thread-per-connection binary, hit 17.3x — with caveats.
The insight matters because hardware accelerators — GPUs, FPGAs, hardware security modules, inference servers — are now on the critical path of online serving, and the standard options for handling the resulting stall are both bad. A context switch costs roughly as much as the offload itself; busy-waiting burns the CPU core. The paper's argument is that servers handling concurrent requests are already suspending and resuming them, so the concurrency machinery needed for overlap ships by default.
The safety asterisk is real: rerouting breaks run-to-completion atomicity, which can create hazards around unlocked shared aggregates. The authors address this with a page-protection detector, validated on stock Redis. Whether that taxonomy holds across messier production codebases is the question this paper cannot answer.