Visual Cache Reuse for Efficient Multimodal Inference

Vision-language model (VLM) inference incurs substantial prefill latency and memory overhead due to the long visual token sequences of large visual inputs. Existing serving systems typically exploit exact cache reuse through prefix caching or position-independent caching, overlooking the potential of approximate caching in real-world workloads. However, directly reusing an approximate cache entry may introduce spurious visual information, which naive approaches that select a subset of important visual tokens for recomputation fail to adequately mitigate this risk. We present VisMemo, an efficient serving system that enables approximate KV cache reuse for modern VLMs. Its core is a system-algorithm co-design that combines an entropy-aware KV cache reuse algorithm with differentiated cache management. Specifically, VisMemo uses patch entropy to identify reuse candidates and warmup attention to prioritize recomputation based on the current query. To support the large cache capacity across GPU memory, CPU memory, and SSD storage, VisMemo applies distinct I/O and cache policies to exact and approximate hits: sparse patch-level I/O for approximate hits, prioritized retention for exact hits, and a reserved CPU victim cache for preset entries selected using offline workload priors and online reuse observations. VisMemo further optimizes the memory access path between the inference engine and the storage backend, eliminating redundant caching and unnecessary data movement. Experiments on multiple public datasets show that VisMemo preserves output quality under approximate reuse. On a real-world production trace, VisMemo increases the overall cache hit rate from 27.78% to 61.45% and reduces prefill latency by up to 48.3% at 8K input tokens.

• 01Low22%

  Heterogeneous Demand-Private Coded Caching: Exact Characterization for Two Users and Two Files

  The paper characterizes the memory-rate trade-off for demand-private coded cachi…
• 02Low21%

  Closing the Calibration Gap in Semantic Caching

  This paper introduces new metrics to evaluate semantic caching systems and shows…
• 03Low20%

  ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding

  This paper presents ZipFusion, a method that enables attention to consume lossle…
• 04Low16%

  TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents

  This paper analyzes the efficiency of AI coding agents and identifies the opport…
• 05Low15%

  FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper

  FUTRANSFORMER is a CUDA implementation of a Llama-style Transformer block for NV…
• 06Low15%

  From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems

  This paper introduces AgentSysBench, a benchmark suite and measurement toolkit f…
• 07Low15%

  DIFFLET: A Fast Inference Engine for Diffusion Transformer Models on AWS Trainium

  The authors present Difflet, a serving engine that accelerates diffusion model d…
• 08Low11%

  SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs

  This paper introduces SPOT-E, a plug-and-play test-time method that improves the…