A new technique called ESTP wants LLM servers to stop guessing how long a response will be and start predicting it.
ESTP, short for Entropy and Semantic Token Pooling, comes from a paper posted to arXiv on August 18, 2026. The problem it targets is mundane but expensive: LLM serving systems often pad every sequence to a fixed maximum length, which wastes compute and slows throughput. Earlier fixes leaned on entropy guided token pooling, using token by token uncertainty as the main signal for predicting output length. ESTP adds a second signal, pulling attention based importance scores straight from the self attention weights computed during the prefill phase, so it can weigh semantically important tokens instead of just noisy ones.
The reuse of prefill activations is the clever part. It means ESTP adds almost no extra memory overhead and only minimal latency, which matters because a length predictor nobody can afford to run is not a length predictor anyone will ship. On the ForeLen benchmark, ESTP beat baseline methods on prediction accuracy and error rate in most scenarios, and paired with a length aware scheduler it improved throughput and cut the padding ratio in end to end tests.
This is an infrastructure paper, not a model paper, and that is exactly why it matters. Every efficiency gain in serving translates directly into lower inference costs, which is the line item every LLM provider is fighting to shrink right now. The catch is the same one every entropy-plus-something paper faces: "in most scenarios" is doing real work in that sentence, and the paper does not spell out where ESTP loses. Worth watching whether independent benchmarks confirm the gains outside ForeLen before serving teams bet infrastructure on it.