A new token-selection method lets vision-language models skip most of the image data they used to process, with barely a dent in accuracy.
Researchers describe StackTok, a training-free technique for trimming the visual token sequences that vision-language models (VLMs) chew through during inference, in a paper posted to arXiv (arXiv:2609.16841v1, https://arxiv.org/abs/2609.16841). Instead of picking tokens by one fixed rule, StackTok switches on the fly between prioritizing tokens relevant to the specific question and tokens that cover more of the image overall, based on how narrow or spread-out the query seems. Tested across five VLMs and ten image-understanding benchmarks, it beat every other training-free selector in the paper's comparisons. On the high-resolution LLaVA-NeXT-7B model, StackTok kept 95.26% of full-token accuracy while using only 160 of 2,880 visual tokens - a 94.4% cut.
That matters because image resolution keeps climbing, and with it the token counts driving up VLM inference costs. A fix that needs no retraining is the cheap kind: it can slot into models already in production instead of forcing a costly do-over, unlike prior methods that bake a fixed relevance-versus-coverage balance in from the start.
Training-free efficiency tricks have a habit of looking great in benchmark tables and less great once they meet real production traffic and hardware quirks. Worth watching for independent replication before anyone bets a deployment on it.