20 results for “Distillation”
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.ⓘ
Want pure semantic search? Try claim verification →
The paper studies the design of context for self-distillation in language models, finding that step-aligned feedback yields the largest gains.
This paper proposes a novel method for training acceleration of speech foundation model distillation to speed up model deployment.
DASH introduces a dual-branch distillation framework to effectively compress class-conditional diffusion models by independently supervising both score branches, significantly preserving guidance fide…
Luyang Fang, Yongkai Chen, Jiazhang Cai, Ping Ma +1 more
The paper proposes Multi-Teacher Bayesian Knowledge Distillation (MT-BKD), a framework that uses Bayesian inference and teacher-informed priors to improve model compression, enhance predictive accurac…
This paper introduces a new distillation reward, the delta signal, for on-policy distillation in reinforcement learning, which captures changes induced by reasoning tuning and improves performance.
This paper proposes a framework called SPOFA to achieve stable knowledge distillation across different architectures.
Gaetan Narozniak, Gérard Biau, Rémi Munos, Ahmad Rammal +1 more
The paper introduces Feedback Distillation, a novel training method that uses a language model's privileged feedback to provide token-level supervision, significantly improving complex reasoning tasks…
This paper re-examines the role of temperature ($ au$) in LLM distillation, demonstrating that while Reverse KL (RKL) is often preferred, Forward KL (FKL) significantly outperforms RKL at higher tempe…
An Zhao, Shengyuan Zhang, Zhongjian Sun, Yixiang Zhou +4 more
This paper proposes a novel distillation framework for flow matching models that enables high-fidelity single-step generation.
The paper proposes a theoretical framework, called constraint-coupled reasoning, to make AI models less susceptible to knowledge distillation by coupling high-level capabilities to internal stability…
Zibo Diao, Jingchu Gai, Xinyue Ai, Zhang Zhang +2 more
The paper introduces Lossless Anti-Distillation Sampling (LADS), a novel sampling scheme that makes harvested data correlated for malicious distillers while ensuring benign users receive statistically…
This paper evaluates the effectiveness of dataset distillation methods and finds that they are comparable to or worse than coreset selection methods on large-scale datasets.
This paper investigates the phenomenon of 'copying' in Distribution Matching Distillation (DMD), finding that high-dimensional distillation causes student models to spontaneously reproduce the teacher…
Ziyang Zheng, Zeju Li, Xiangyu Wen, Jianyuan Zhong +4 more
The paper reframes context distillation as a latent memory management problem, proposing a modular framework using LoRA adapters and a Self-Gating mechanism for efficient, selective memory retrieval a…
The paper develops a quantitative framework to analyze and improve flow distillation in diffusion models, providing stability guarantees and suggesting non-uniform time scheduling to reduce approximat…
Yiru Yang, Junling Wang, Nishant Kumar Singh, Luohong Wu +1 more
The paper proposes a novel layer and point-wise projection mapping combined with LoRA injection to efficiently distill knowledge from a large teacher model to a small student model, significantly impr…
Jaeeun Baik, Ui-Hyeop Shin, Jiwoon Lee, Woocheol Jeong +1 more
The paper proposes DASH, a self-distillation framework that improves ASR robustness in noisy environments by learning clean-noisy consistency from paired views.
This paper presents DisaggKD, a knowledge distillation framework that separates teacher and student models across nodes to overcome per-GPU memory limits.