20 results for βserving systemβ
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.β
Want pure semantic search? Try claim verification β
Zhang Cao, Shujie Han, Juncheng Zhang, Yuanming Ren +2 more
This paper presents LUMEN, a fault-tolerant LLM serving system that improves serving and recovery times by treating recovery as a load-aware coordination problem.
Yunfei Feng, Jingkai He, Shiqi Liu, Cheng Zhang +2 more
The paper presents FastEEP, a system for fast intra-instance scaling of Mixture-of-Experts (MoE) models in production.
Hetvi Shastri, Pragya Sharma, Walid A. Hanafy, David Irwin +2 more
This paper presents FMplex, a serving system that shares foundation model backbones across multiple tasks to reduce latency and increase task hosting capacity.
Sookyung Choi, Seungyong Lee, Kangkyu Park, Yunseo Chun +10 more
This paper presents NELSSA, a serving system that integrates GPUs with Processing-near-Memory (PNM) devices to efficiently handle mixed-length workloads in LLMs, achieving up to 5.5x decode throughputβ¦
Youhe Jiang, Haoxu Wang, Haotong Bao, Kai Jiang +4 more
TurboServe is a serving system designed for streaming video generation workloads that reduces latency and cost.
Weinan Liu, Zeyuan Ding, Dian Ding, Chengcheng Wan +4 more
This paper presents SliceScheduler, a dynamic operator-level scheduling system for multi-tenant model serving that improves GPU utilization and maintains SLA.
Chaokun Chang, Yukun Zhou, Kaihua Fu, Dakai An +9 more
This paper introduces AgentSysBench, a benchmark suite and measurement toolkit for agentic applications, and identifies six properties that distinguish agentic workloads from conventional LLM serving.
Hanzhi Zhang, Anxuan Che, Hang Qi, Qiyue Li +1 more
One clear sentence describing what this paper does
Dilina Dehigama, Shyam Jesalpura, Zeyu Xu, Marton Nemeth +3 more
The paper introduces Spandana, an architecture that decouples SLO enforcement from cost optimization in cloud-based online services, achieving high utilization, strict SLO adherence, and cost savings.
Jiayi Qian, Zishen Wan, Hanchen Yang, Chun Tao +2 more
The paper presents Dyserve, a workflow-aware serving layer for agentic AI applications that compiles per-node model and verifier choices into an integer linear program, allowing for efficient model seβ¦
A simple online algorithm is presented for the stacking problem to avoid shifts with a sufficient condition involving stacking area dimension, load/unload points, and maximum items.
This paper studies software aging in GPU-based LLM serving systems using a 216-hour campaign.
This paper studies the maximum number of jobs a queueing system can serve covertly without being detected.
This paper introduces MARS, a framework for repeat-order food delivery recommendation that combines pre-trained language models with collaborative retrieval and contextual filtering.