This paper presents NELSSA, a serving system that integrates GPUs with Processing-near-Memory (PNM) devices to efficiently handle mixed-length workloads in LLMs, achieving up to 5.5x decode throughput…

The paper introduces Rotary GPU, an exploratory execution approach demonstrating that large Mixture-of-Experts models can be run locally on consumer GPUs with limited VRAM, achieving usable decode thr…

This paper investigates the energy efficiency of Large Language Model (LLM) serving using emerging memory technologies, specifically monolithic 3D (M3D) integration, and presents simulation results sh…

Lodestar is a novel online learning-based request routing system that significantly improves LLM inference efficiency by dynamically assigning incoming requests to the optimal GPU instance to minimize…

This paper presents a comprehensive measurement study on the performance and energy consumption of large language models on mobile devices, using five frameworks and three hardware backends, and intro…

This paper proposes TileLens, a system to mitigate read amplification in Large-Granularity Memory Systems (LGMS) for Large Language Model (LLM) inference by adopting a tile-major layout.

This study benchmarks four local LLMs for natural-language-to-SQL querying in biopharma manufacturing, finding that general-purpose code-tuned models like Llama 3.1 8B and Qwen 2.5 Coder 7B outperform…

This paper evaluates the performance of large language models on popular benchmarks and finds that only a small percentage of the performant implementations are significantly faster than canonical sol…

The paper proposes PagedWeight, a method for managing Mixture-of-Experts (MoE) language model serving in KV-cache-intensive scenarios, achieving FP16-equivalent accuracy with up to 72.0% GPU memory sa…

The paper proposes the Communicating Finite Automata (CFA) abstraction and a framework for component program refactoring in large language model inference services, achieving significant speedup and r…

This paper proposes 3DLS, a 3D-stacked chiplet architecture for large language model serving, which separates traffic classes and achieves higher throughput and lower latency than conventional archite…

The paper presents a field study on deploying large inference workloads on a non-GPU AI accelerator and identifies eight categories of limitations.

Aquifer is the first system to serve MicroVM snapshots from a hierarchical CXL+RDMA memory pool, achieving 2.2x geometric-mean speedup in end-to-end invocation time over Firecracker.

This paper presents Festina, a profiling-guided, power-aware control plane for minimizing energy consumption in serverless large language model (LLM) serving.

This paper characterizes the performance of vision-language model inference on the Qualcomm SM8750 using FastVLM-0.5B as a case study, showing significant speedups and energy savings for different pha…

The paper systematically analyzes the benefits and limits of Attention-FFN Disaggregation (AFD) for Mixture-of-Experts (MoE) LLM serving, demonstrating that AFD is crucial for achieving high throughpu…

This paper explores methods to reduce latency in GPU collective communications for large language model inference, achieving near-optimal designs with barrier-free synchronization and efficient use of…