Streaming video understanding models face a fundamental challenge: they must answer queries at any point during an ongoing stream, using only what they have observed so far, while operating under fixed memory and computation budgets. Existing approaches add memory banks, retrieval modules, or visual token compression to preserve long-range history. However, strong recent-window baselines have shown that indiscriminate history injection can dilute current-scene perception. According to a research paper published on arXiv, the key challenge is not whether to use memory, but how to allocate it selectively.
The Problem: Budgeted Online Latent Evidence Allocation
The authors — Ge, Haonan, Wang, Yiwei, Wu, Hang, and Cai, Yujun — formulate this as budgeted online latent evidence allocation. They propose a framework called SelectStream, a selective latent-memory framework that keeps the current observation directly visible to a frozen Vision-Language Model (VLM) while exposing historical information only through a compact, query-conditioned evidence budget. This design avoids replaying frames or growing the context with stream length, making it suitable for real-time applications.
SelectStream's Three Coordinated Mechanisms
SelectStream employs three coordinated mechanisms to govern when to write, what to preserve, and how to retrieve historical information:
- Surprise-driven adaptive windowing: Determines which moments in the stream are worth remembering based on novelty or deviation from expected patterns.
- Priority-preserving consolidation: Ensures that important historical details are retained in a fixed-capacity latent memory graph without being overwritten by less relevant information.
- Query-conditioned graph reasoning: Retrieves relevant evidence from the memory graph based on the current query, injecting it as latent tokens for answer generation.
Retrieved evidence is calibrated and then injected as latent tokens into the frozen VLM for answer generation. The system does not replay frames or expand the context window as the stream lengthens, keeping computation constant.
Experimental Results
The paper reports that SelectStream achieves strong online streaming performance while preserving general video understanding. The specific results are:
| Benchmark | Score |
|---|---|
| StreamingBench | 82.67% |
| OVO-Bench | 67.03% |
| Offline video benchmarks (average) | 74.4% |
SelectStream outperforms strong recent-window baselines and prior streaming memory methods on these metrics.
Implications for Streaming Video Applications
Because SelectStream is designed to operate under fixed memory and computation budgets while answering queries at any moment during an ongoing stream, it is directly applicable to use cases such as live surveillance, autonomous driving, and real-time video analytics. The framework demonstrates that selective memory allocation, rather than exhaustive history retention, can improve both efficiency and accuracy. The paper is available on arXiv under a Creative Commons Attribution 4.0 International license.
