Lodestar is a novel online learning-based request routing system that significantly improves LLM inference efficiency by dynamically assigning incoming requests to the optimal GPU instance to minimize…

This paper proposes CrossPool, a serving engine for cold Machine Learning Models (LLMs) that separates weights and KV-cache into two GPU memory pools to improve GPU memory utilization and long-context…

This paper presents Festina, a profiling-guided, power-aware control plane for minimizing energy consumption in serverless large language model (LLM) serving.

This paper proposes 3DLS, a 3D-stacked chiplet architecture for large language model serving, which separates traffic classes and achieves higher throughput and lower latency than conventional archite…

This paper proposes TileLens, a system to mitigate read amplification in Large-Granularity Memory Systems (LGMS) for Large Language Model (LLM) inference by adopting a tile-major layout.

The paper characterizes 'dead-entry' TLB misses in GPUs, which occur when recently evicted translations are immediately re-walked, and proposes DEPOT, a Bloom filter mechanism that significantly reduc…

This paper explores methods to reduce latency in GPU collective communications for large language model inference, achieving near-optimal designs with barrier-free synchronization and efficient use of…

KernelFlume is a decode-centric architecture that disaggregates the stable projection/FFN path from core-attention computation to improve efficiency and reduce cost in serving long-context demand.

PIMID is an execution- and trace-driven full-system simulator for Processing-in-Memory systems, supporting multiple memory technologies, execution models, and placement of processing elements.

The paper introduces Rotary GPU, an exploratory execution approach demonstrating that large Mixture-of-Experts models can be run locally on consumer GPUs with limited VRAM, achieving usable decode thr…

This paper proposes Gleam, a framework for efficient GPU sharing across local-area CUDA devices, reducing bandwidth overhead, improving API call latency, and ensuring context consistency.

A proactive scheduler is proposed to reduce interference between prefill and decode phases in disaggregated large language model serving, improving P95 Time-to-First-Token and SLO attainment.

Aquifer is the first system to serve MicroVM snapshots from a hierarchical CXL+RDMA memory pool, achieving 2.2x geometric-mean speedup in end-to-end invocation time over Firecracker.

The paper introduces Spandana, an architecture that decouples SLO enforcement from cost optimization in cloud-based online services, achieving high utilization, strict SLO adherence, and cost savings.

The paper proposes a fine-grained approach to overlap expert compute with the second all-to-all communication in Mixture-of-Experts (MoE) models, improving distributed MoE execution efficiency on mult…

The paper introduces COSM, a cooperative scheduling framework to facilitate concurrent operation of Processing-in-Memory (PIM) and CPU tasks on mobile platforms, improving PIM throughput by up to 2.8x…

This paper investigates the energy efficiency of Large Language Model (LLM) serving using emerging memory technologies, specifically monolithic 3D (M3D) integration, and presents simulation results sh…

This paper presents a comprehensive measurement study on the performance and energy consumption of large language models on mobile devices, using five frameworks and three hardware backends, and intro…