~ similar to 2606.19271· 20 results
Zhuoren Ye, Tianyu Wo, Dinghao Xue, Mingming Zhang +3 more
This paper proposes CrossPool, a serving engine for cold Machine Learning Models (LLMs) that separates weights and KV-cache into two GPU memory pools to improve GPU memory utilization and long-context…
Xinwei Qiang, Yifan Hu, Shixuan Sun, Jing Yang +5 more
GF-DiT: A policy-programmable runtime for elastic DiT serving
This paper presents a measurement protocol and benchmarking harness for evaluating the performance of different multi-node strategies for submitting tasks on shared Slurm clusters, focusing on their i…
This paper presents Festina, a profiling-guided, power-aware control plane for minimizing energy consumption in serverless large language model (LLM) serving.
Weinan Liu, Zeyuan Ding, Dian Ding, Chengcheng Wan +4 more
This paper presents SliceScheduler, a dynamic operator-level scheduling system for multi-tenant model serving that improves GPU utilization and maintains SLA.
Jiayi Qian, Zishen Wan, Hanchen Yang, Chun Tao +2 more
The paper presents Dyserve, a workflow-aware serving layer for agentic AI applications that compiles per-node model and verifier choices into an integer linear program, allowing for efficient model se…
This paper studies software aging in GPU-based LLM serving systems using a 216-hour campaign.
This paper introduces execution-state capsules for low-latency, small-batch, on-device AI serving.
Yuyang Zhao, Yicheng Pan, Qiyuan He, Jincheng Yu +5 more
SANA-Streaming introduces a novel, efficient framework that enables real-time, high-resolution streaming video-to-video editing by combining a hybrid diffusion transformer with specialized training an…
Kunxiong Zhu, Zhihao Shu, Hangyu Zheng, Minghai Qin +3 more
EAServe optimizes and improves the performance of multimodal LLMs by repositioning Encode as the control point of the EPD pipeline
Yuncheng Yao, Zhenzhou Qi, Junyao Zheng, Chung-Hsuan Tung +2 more
This paper presents Beaver, a GPU sharing system that jointly manages compute and memory resources to protect the vRAN's strict processing deadline while serving AI workloads.
Shushu Yi, Wenbo Wu, Guoci Chen, Junrong Zhu +5 more
GNStor is a GPU-native AFA system that enables direct GPU access to remote AFA without CPU intervention.
Sookyung Choi, Seungyong Lee, Kangkyu Park, Yunseo Chun +10 more
This paper presents NELSSA, a serving system that integrates GPUs with Processing-near-Memory (PNM) devices to efficiently handle mixed-length workloads in LLMs, achieving up to 5.5x decode throughput…
The paper introduces Rotary GPU, an exploratory execution approach demonstrating that large Mixture-of-Experts models can be run locally on consumer GPUs with limited VRAM, achieving usable decode thr…
Coloc is a storage-management layer that optimizes cross-modal access locality in distributed vector stores.
Chunan Shi, Yilei Chen, Yilin Chen, Xupeng Miao +1 more
The paper proposes AsymCache, a computation-latency-aware KV cache management system that optimizes LLM inference by aligning cache eviction decisions with GPU attention kernel performance, significan…