A new inference simulator predicts LLM performance kernel by kernel, potentially replacing the slow, deployment-specific GPU benchmarks that keep ML engineers waiting.
KernelSight-LM decomposes each inference step into three sub-models: a roofline kernel model with a learned efficiency term, a communication model, and a host-overhead model, coordinated by a discrete-event scheduler that also handles prefix caching and continuous batching. The tool ships in two tiers. The cross-generation tier relies only on hardware specifications and microbenchmarks from previously profiled GPUs, predicting per-kernel latency on an unseen GPU to 12.1% error, 1.8 times better than a plain roofline baseline. The target-measured tier adds a single microbenchmark sweep on the actual hardware, sharpening per-kernel error to 3.8%, a 7.3x improvement over a comparable baseline.
The 3.8% figure is per-kernel accuracy, not the metric practitioners actually stress-test against. End-to-end time-to-first-token (TTFT), the latency number that determines whether a deployment meets its SLA, carries errors of 15.4% in the cross-generation tier and 14.3% in the target-measured tier. Throughput predictions land around 3% error in both configurations, and time-per-output-token (TPOT) sits at 12.8% and 6.2% respectively, so the tool is more reliable for capacity planning than for tight latency commitments.
The paper covers six model families, which is encouraging, but whether a 15% TTFT miss fits your deployment budget depends entirely on what you are building.