This paper presents a comprehensive measurement study on the performance and energy consumption of large language models on mobile devices, using five frameworks and three hardware backends, and intro…

This paper proposes 3DLS, a 3D-stacked chiplet architecture for large language model serving, which separates traffic classes and achieves higher throughput and lower latency than conventional archite…

This paper proposes CrossPool, a serving engine for cold Machine Learning Models (LLMs) that separates weights and KV-cache into two GPU memory pools to improve GPU memory utilization and long-context…

This paper proposes TileLens, a system to mitigate read amplification in Large-Granularity Memory Systems (LGMS) for Large Language Model (LLM) inference by adopting a tile-major layout.

This paper presents Festina, a profiling-guided, power-aware control plane for minimizing energy consumption in serverless large language model (LLM) serving.

The paper introduces COSM, a cooperative scheduling framework to facilitate concurrent operation of Processing-in-Memory (PIM) and CPU tasks on mobile platforms, improving PIM throughput by up to 2.8x…

Proposed EnerInfer framework manages energy efficiency, throughput, and thermal comfort for on-device LLM inference, improving energy efficiency up to 65% without QoE violation.

The paper proposes AsymCache, a computation-latency-aware KV cache management system that optimizes LLM inference by aligning cache eviction decisions with GPU attention kernel performance, significan…

This paper presents NELSSA, a serving system that integrates GPUs with Processing-near-Memory (PNM) devices to efficiently handle mixed-length workloads in LLMs, achieving up to 5.5x decode throughput…

Aquifer is the first system to serve MicroVM snapshots from a hierarchical CXL+RDMA memory pool, achieving 2.2x geometric-mean speedup in end-to-end invocation time over Firecracker.

PIMID is an execution- and trace-driven full-system simulator for Processing-in-Memory systems, supporting multiple memory technologies, execution models, and placement of processing elements.

The paper introduces Chimera, a highly efficient and scalable MCU designed for ultra-low-power edge AI inference, achieving 3.1 TOPS/W by integrating a dedicated transformer accelerator and a QoS-guar…

The paper characterizes 'dead-entry' TLB misses in GPUs, which occur when recently evicted translations are immediately re-walked, and proposes DEPOT, a Bloom filter mechanism that significantly reduc…

A proactive scheduler is proposed to reduce interference between prefill and decode phases in disaggregated large language model serving, improving P95 Time-to-First-Token and SLO attainment.

Lodestar is a novel online learning-based request routing system that significantly improves LLM inference efficiency by dynamically assigning incoming requests to the optimal GPU instance to minimize…

The paper proposes PagedWeight, a method for managing Mixture-of-Experts (MoE) language model serving in KV-cache-intensive scenarios, achieving FP16-equivalent accuracy with up to 72.0% GPU memory sa…

The paper systematically analyzes the benefits and limits of Attention-FFN Disaggregation (AFD) for Mixture-of-Experts (MoE) LLM serving, demonstrating that AFD is crucial for achieving high throughpu…

Physical AI inference (batch-1 decode) is primarily memory-bandwidth-bound, but the observed latency gap between fast and slow GPUs is not solely due to memory bandwidth, as launch-side overheads beco…