The paper introduces Rotary GPU, an exploratory execution approach demonstrating that large Mixture-of-Experts models can be run locally on consumer GPUs with limited VRAM, achieving usable decode thr…

The paper proposes a fine-grained approach to overlap expert compute with the second all-to-all communication in Mixture-of-Experts (MoE) models, improving distributed MoE execution efficiency on mult…

Lodestar is a novel online learning-based request routing system that significantly improves LLM inference efficiency by dynamically assigning incoming requests to the optimal GPU instance to minimize…

This paper proposes TileLens, a system to mitigate read amplification in Large-Granularity Memory Systems (LGMS) for Large Language Model (LLM) inference by adopting a tile-major layout.

The paper systematically analyzes the benefits and limits of Attention-FFN Disaggregation (AFD) for Mixture-of-Experts (MoE) LLM serving, demonstrating that AFD is crucial for achieving high throughpu…

The paper presents a field study on deploying large inference workloads on a non-GPU AI accelerator and identifies eight categories of limitations.

This survey systematically reviews resource consumption threats in large language models (LLMs) to provide a unified view of the problem landscape, from threat induction to mitigation.

The paper proposes PagedWeight, a method for managing Mixture-of-Experts (MoE) language model serving in KV-cache-intensive scenarios, achieving FP16-equivalent accuracy with up to 72.0% GPU memory sa…

This paper presents a method for compressing matrices using a RePair straight-line program (SLP), allowing matrix-vector products with time and space proportional to the compressed size, and demonstra…

The paper proposes AsymCache, a computation-latency-aware KV cache management system that optimizes LLM inference by aligning cache eviction decisions with GPU attention kernel performance, significan…

Omni-Flow is a distributed scheduling framework for multimodal inference that provides a unified abstraction for control flow, data flow, and compute flow.

This paper evaluates the performance of large language models on popular benchmarks and finds that only a small percentage of the performant implementations are significantly faster than canonical sol…

This paper presents DOPS, a hardware-aware framework for optimizing operator scheduling and weight layouts in Large Language Models, achieving significant speedups over prefill-decode disaggregation.

Taurus is a single-machine system for efficient Graph Neural Network (GNN) inference on large-scale graphs that do not fit in RAM, using source-centric broadcasts and a pipelined GPU-CPU-SSD hierarchy…

Hawkeye is a system that allows perfect, precision-preserving reproduction of GPU-level matrix multiplication operations on a CPU, enabling efficient and trustworthy third-party auditing of machine le…

This paper proposes a new architecture for transformer-based language models called 'former', which allocates capacity non-uniformly across network depth by maintaining wider early and late layers whi…