This paper presents HiState, a prefix cache management system for hybrid models serving agent workloads to reduce inference costs.
HiState introduces a novel hierarchical approach to prefix caching that balances checkpoint storage and reclamation.
Before reading this…
Applications
To understand this paper, make sure you know these concepts first:
Hybrid large language models combine full-attention and recurrent linear layers to reduce inference costs. Prefix caching can further reduce redundant computation across requests, but a recurrent checkpoint cannot recover an earlier state corresponding to a shorter prefix. Existing hybrid prefix caching systems retain checkpoints at selected boundaries to support common reuse patterns in conventional conversations. However, agent workloads modify and revisit earlier contexts, exposing reusable prefixes whose checkpoints were never saved or have already been evicted. A straightforward approach is to retain more checkpoints to reduce recomputation, but their cumulative footprint can displace reusable prefixes from other sessions and cause additional cache misses. To address these problems, we present HiState, a prefix cache management system for hybrid models serving agent workloads. Our key observation is that prefix reuse accesses all matching key-value (KV) entries but only one recurrent checkpoint, regardless of how many checkpoints are retained. This asymmetry allows HiState to retain historical checkpoints on SSD while prioritizing fast memory for KV entries: each hit retrieves only one checkpoint whose size is independent of context length. Within this hierarchy, we propose adaptive interval control to balance checkpoint storage against recomputation, and unified eviction to coordinate checkpoint sparsification with reclamation of stale trajectories. We implement HiState on top of vLLM and evaluate it on two public coding-agent trace datasets and controlled microbenchmarks. On real agent traces, HiState achieves up to 5.73x and 5.44x speedups in p99 TTFT over vLLM-Marconi and SGLang-HiCache, respectively.
Visual Cache Reuse for Efficient Multimodal Inference
VisMemo is a serving system that enables approximate KV cache reuse for modern v…
SPAR: Stepwise Personalized Autoregressive Recommendation with Prefix-Balanced Semantic IDs
This paper proposes a generative recommendation framework called SPAR that impro…
Improving Long-Context Retrieval with Multi-Prefix Embedding
The paper proposes Multi-Prefix Embedding (MPE), a method that encodes documents…
CXLResume: Retained-State Checkpoint and Restore for In-Memory Services
CXLResume reduces checkpoint and restore time by retaining shared application st…
DIFFLET: A Fast Inference Engine for Diffusion Transformer Models on AWS Trainium
The authors present Difflet, a serving engine that accelerates diffusion model d…
ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding
This paper presents ZipFusion, a method that enables attention to consume lossle…
Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse
This paper proposes a method to improve multimodal agent performance by reusing…
TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
This paper presents TAO, a tool-aware joint scheduling framework for LLM-based a…