Built with and by Teycir Ben Soltane•
How to Use•FAQ•GitHub•arXiv.org•
Share:
ArXivCSExplorer
☆☆Bookmarks🏆RSSHow to UseFAQ
Home/2610.34731
cs.AREmpirical

MCF: A Memory- and Communication-Aware Co-Designed Framework for Long-Context LLM Inference

Huaizhi Zhang, Jiacheng Zhu, Xuqi Zhu, Junkyu Lee, Klaus McDonald-Maier, Xiaojun Zhai
Oct 26, 2026
PDF HTML
AI Summaryllama-3.1-8b-instruct

MCF optimizes long-context inference through communication-aware and stage-adaptive KV-cache layout transitions.

MCF presents a novel framework that combines memory-aware and communication-aware techniques to optimize long-context inference.

Keywords

Long-context inferenceMemory-awareCommunication-awareKV-cacheFPGAHardware-software co-design

Before reading this…

Knowledge of computer architecture and distributed systemsUnderstanding of long-context inference

Applications

  • →Natural language processing
  • →Machine translation
Skill Ladder

To understand this paper, make sure you know these concepts first:

  • Knowledge of computer architecture and distributed systemsfind papers →
  • Understanding of long-context inferencefind papers →

Abstract

More Like This

This paper presents Memory- and Communication-Aware Framework for Long-Context LLM Inference (MCF) that optimizes long-context inference through communication-aware and stage-adaptive KV-cache layout transitions. At the core of MCF is {Memory-Aware Replication and Scheduling} (MARS), a dynamic orchestration mechanism for chunked prefill. MARS exploits early-stage memory headroom by selectively replicating highly reused KV blocks, improving locality and hiding KV propagation latency behind attention computation. As prefill progresses and memory pressure increases, MARS progressively drains redundant replicas and converges the KV cache into a sharded decode-ready layout, avoiding expensive layout conversion at the prefill--decode boundary. For decoding, MCF adopts a query-centric execution model in which KV shards remain stationary while lightweight query tensors are routed across workers. This changes communication from KV-cache-dependent transfer to query-dependent transfer, reducing communication growth with context length. MCF further integrates a hardware-software co-designed execution layer based on an FPGA compute-NIC datapath, enabling metadata-driven KV routing and collective operations directly in the network path. Experiments show that MCF improves TTFT, throughput, and scalability over state-of-the-art distributed inference systems, with larger benefits as context length and cluster size increase.

Paper Label
TypeEmpirical
NoveltyNotable
Related Papers
  • 01Low19%

    From Memory Hits to Answerable Evidence: Query-Conditioned Memory Compilation for Long-Term Agent Me…

    This paper proposes MemCompile, a query-conditioned evidence compilation framewo…

  • 02Low18%

    SPARTA: A Sparse Ternary LLM Accelerator with Adaptive Attention and Unified Datapath for Long-Conte…

    This paper presents SPARTA, a sparse ternary LLM accelerator for efficient long-…

  • 03Low16%

    Visual Cache Reuse for Efficient Multimodal Inference

    VisMemo is a serving system that enables approximate KV cache reuse for modern v…

  • 04Low