Video-MemComp: Taylor-Guided Sensitivity-Aware Memory Management for Streaming Video Understanding

Yonghui Niu1,  Yuhao Ji1,  Chang Nie1,  Jiao Zhang2,  Yilong Li2,  Junlan Feng2,  Ran He3,  Caifeng Shan1,  Chaoyou Fu1✉
1 Nanjing University 2 China Mobile Research Institute 3 Institute of Automation, CAS
arXiv GitHub

Abstract

Multimodal Large Language Models (MLLMs) are currently hindered by severe memory bottlenecks when processing streaming videos, as the KV cache grows linearly with sequence length. Existing compression methods often rely on heuristic eviction strategies, failing to account for the explosive accumulation of approximation errors during long-context generation.

In this paper, we propose Video-MemComp, a novel hybrid framework that achieves extreme $O(1)$ memory compression. Our method theoretically decomposes the compression error into two orthogonal components — local approximation error $\mathcal{E}_l$ and layer-wise structural sensitivity $\mathcal{S}_l$ — derived via a first-order Taylor expansion. To minimize $\mathcal{E}_l$, we introduce a pre-RoPE token aggregation strategy incorporating an attention-conserving logit bias. To manage $\mathcal{S}_l$, we dynamically allocate KV-cache budgets based on offline gradient priors and online context entropy.

Extensive experiments demonstrate that Video-MemComp consistently achieves performance on par with or superior to full-cache baselines across MLVU and VideoMME. Specifically, our framework improves the overall accuracy of Qwen2.5-VL-3B from 62.3% to 64.1% while maintaining a fixed KV-cache footprint of only 6,000 tokens per layer, effectively achieving over 90% memory savings without performance degradation.

Method

Guided by a first-order Taylor expansion, we decompose the global memory compression problem into two orthogonal optimization objectives. The global output error is bounded as $\|\Delta\mathcal{O}\| \le \sum_{l} \mathcal{S}_l \cdot \mathcal{E}_l$, leading to a hierarchical design that jointly addresses intra-layer token compression and inter-layer budget scheduling.

Figure 1: Memory complexity comparison

Figure 1. Memory complexity comparison. While standard Video-LLM models suffer from linear O(T) memory growth as sequence length increases, Video-MemComp maintains a strictly O(1) memory footprint via a Taylor expansion-guided hybrid compression strategy.

Figure 2: Framework overview

Figure 2. Overview of the Video-MemComp framework. Guided by a first-order Taylor expansion, the pipeline minimizes local approximation error via Pre-RoPE token aggregation with an attention-conserving logit bias, and manages layer-wise structural sensitivity via a Tri-Hybrid budget allocation strategy combining offline gradient priors and online context entropy.

Experiment Results

1. Comprehensive Offline Video Understanding

We evaluate Video-MemComp across three representative offline benchmarks: VideoMME, MLVU, and EgoSchema. As shown in Table 1, our method consistently meets or exceeds the performance of full-cache baselines while using significantly less memory. This demonstrates that our Taylor expansion-guided strategy effectively identifies and preserves semantically critical information while discarding temporal redundancy.

Table 1: Offline benchmark results

Table 1. Performance comparison on offline video understanding benchmarks (VideoMME, MLVU, EgoSchema).

2. Robustness Across Memory Budgets

Figure 2 illustrates the performance curve on MLVU using the Qwen2-VL-7B backbone across varying KV cache budgets. Video-MemComp outperforms all strong baselines, including SnapKV and InfiniPot, across the entire budget spectrum. Notably, at 6K, 12K, and 24K token budgets, Video-MemComp surpasses the Full-KV baseline, demonstrating that intelligent compression can outperform naive full-cache retention.

Figure 2: Accuracy vs KV cache budget

Figure 2. Accuracy versus KV cache budget on MLVU (Qwen2-VL-7B backbone). Video-MemComp maintains a consistently superior accuracy–memory trade-off compared to all baselines.

3. Streaming Understanding & System Efficiency

Evaluated using the LLaVA-OneVision-7B backbone on real-world streaming benchmarks, Video-MemComp achieves the highest average accuracy among all memory-constrained methods. As detailed in Table 3, our framework reduces Peak GPU Memory by ~27% and cuts Time-to-First-Token (TTFT) by over 50% relative to the ReKV baseline, confirming its suitability for latency-sensitive deployment scenarios.

Table 2: Streaming benchmark results

Table 2. Quantitative results on RVS streaming benchmarks using LLaVA-OneVision-7B.

Table 3: System efficiency

Table 3. System efficiency analysis: peak memory and TTFT on LLaVA-OneVision-7B.

Citation

If you find this work useful for your research, please consider citing:

@article{niu2024videomemcomp,
  title   = {Video-MemComp: Taylor-Guided Sensitivity-Aware Memory Management for Streaming Video Understanding},
  author  = {Niu, Yonghui and Ji, Yuhao and Nie, Chang and Zhang, Jiao and
             Li, Yilong and Feng, Junlan and He, Ran and Shan, Caifeng and
             Fu, Chaoyou},
  journal = {arXiv preprint},
  year    = {2024}
}