20 results for “Understanding of Mixture-of-Experts models, Distributed systems, Edge computing, Queueing theory”
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.ⓘ
Want pure semantic search? Try claim verification →
Bingcong Li, Hao Ma, Michael Muehlebach, Niao He +1 more
This paper proposes MirrorMoE to reconcile the exploration dilemma in mixture-of-experts models by shrinking the exploration space through token-adaptive suppression of unnecessary experts.
The paper develops a minimal dynamical model showing that adaptive softmax routing in Mixture-of-Experts (MoE) layers can undergo abrupt transitions to load imbalance via bifurcation mechanisms.
Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao, Semih Yavuz +2 more
This paper presents a memory-efficient training stack for Mixture-of-Experts (MoE) models, combining and specializing parallelism techniques for maximal efficiency.
John Stephan, Razvan Puscasu, Xu Yong, Ding Jiutao +1 more
This paper presents INC-MoE, an expert-parallel MoE layer that offloads dispatch and combine operations to the network switch to reduce latency and increase throughput.
The paper introduces ProbMoE, a probabilistic routing framework that tackles the non-differentiability of top-$k$ routing in Mixture-of-Experts (MoE) models, achieving strong performance with improved…
Jiarui Feng, Hanqing Zeng, Karish Grover, Ruizhong Qiu +10 more
The paper proposes DAG-MoE, a novel sparse Mixture-of-Experts framework that replaces standard weighted-sum aggregation with structural aggregation to enhance model performance and enable multi-step r…
Lorenzo Sani, Zeyu Cao, Meghdad Kurmanji, Alex Iacob +4 more
This paper introduces FoMoE, a system that reduces communication costs and achieves speedups in training large language models by partitioning expert layers across workers.
The paper proposes a fine-grained approach to overlap expert compute with the second all-to-all communication in Mixture-of-Experts (MoE) models, improving distributed MoE execution efficiency on mult…
This paper proposes a new router redesign for Mixture-of-Experts models using Manifold Power Iteration to align router rows with the principal singular directions of associated experts.
Yunfei Feng, Jingkai He, Shiqi Liu, Cheng Zhang +2 more
The paper presents FastEEP, a system for fast intra-instance scaling of Mixture-of-Experts (MoE) models in production.
Zhiyao Xu, Aoxue Liu, Zhanjie Ding, Dan Zhao +2 more
The paper proposes Task-Aware Coactivation Grouping (TACG) to significantly reduce communication costs in multi-task MoE inference by grouping experts based on task-specific co-activation patterns, ou…
The paper analyzes a new class of asynchronous adaptive first-order optimization methods and proves their stochastic convergence rate is O(1/sqrt{t}) for non-convex functions.
The paper proposes GC-MoE, a graph-conditioned Mixture of Experts framework, to improve traffic forecasting by assigning personalized, specialized forecasting experts to individual road segments.
Jithin S., Roshin Sleeba C., Anvin Mariya P. B., Asmitha K. A. +3 more
A unified multi-task malware analysis framework based on Mixture of Experts (MoE) architectures is proposed for malware family classification, packed versus unpacked detection, and malware versus beni…
Xin Yuan, Ning Li, Wenchao Xu, Athanasios V. Vasilakos +2 more
This paper proposes HetRoute, a framework for optimizing Mixture-of-Experts model inference over geo-distributed heterogeneous edge servers by considering jointly the effects of cross-server link band…
This paper proposes a proactive fair scheduling framework to prevent fabric oversubscription and eliminate incast in Mixture of Experts (MoE) architectures, demonstrating consistent link utilization a…
Ziyu Huang, Yangjie Zhou, Chenhao Zhu, Zihan Liu +12 more
This paper presents Weave, a system that improves the performance of Mixture-of-Experts (MoE) inference by dynamically scheduling GPU resources.
En-Ming Huang, An-Cheng Chang, Bai-Cheng Jeng, Shih-Hao Hung +1 more
This paper introduces a distribution-aware framework for modeling and benchmarking Mixture-of-Experts (MoE) inference, showing that the best fused-MoE kernel changes with routing skew and token count,…