A new memory scheme lets AI models watch long videos without blowing through their token budget.
Researchers introduce Prefix-Steered Recurrent Memory, or PREM, a framework built for frozen vision-language models. It splits video processing into two stages: a recurrent writer compresses the video stream into a compact 256 KiB multi-slot memory, then a question-conditioned readout steers the model's existing key-value cache during prefill instead of appending new tokens. That write-once, query-many design was tested across six long-video benchmarks in both offline and streaming settings. Under a tight 16-frame budget, it lifted macro-average accuracy by 3.06% on Qwen2.5-VL-3B, with an 11.0% jump on action antonym identification and 9.9% on localized needle retrieval, while tuning only 0.24% of the model's parameters and adding just 0.03 GiB of peak GPU memory.
Most long-video fixes so far have meant compressing frames, bolting on memory tokens, or rewriting the model's internal KV cache directly - approaches that either throw away detail or make every new question expensive. PREM's split between ingesting the video once and querying it repeatedly is the more interesting move here: the same compact memory state can field many different questions without re-reading the video or lengthening the prompt each time, which matters for anything latency-sensitive, like video search or streaming review tools.
The catch is that all of this is demonstrated on one 3-billion-parameter model. Whether the gains survive at larger scale, or against the closed video-understanding systems built by bigger labs, is still an open question.