ArXivCSExplorer
☆☆Bookmarks🏆RSSHow to UseFAQ
Built with and by Teycir Ben Soltane•
How to Use•FAQ•GitHub•arXiv.org•
Share:

20 results for “Distillation”

CS papers only

Hybrid search: Keyword + semantic, ranked by combined score.ⓘ

Want pure semantic search? Try claim verification →

cs.AIcs.LGEmpiricalRecentJun 9, 2026

The Role of Feedback Alignment in Self-Distillation

Semih Kara, Oğuzhan Ersoy

The paper studies the design of context for self-distillation in language models, finding that step-aligned feedback yields the largest gains.

View →
eess.AScs.AIcs.CLEmpiricalRecentJun 10, 2026

Fast Speech Foundation Model Distillation Using Interleaved Stacking

Eungbeom Kim, Kyogu Lee

This paper proposes a novel method for training acceleration of speech foundation model distillation to speed up model deployment.

View →
cs.CVcs.AIcs.LGRecentMay 30, 2026

DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models

Abdullah Al Shafi, Kazi Saeed Alam, Sk Imran Hossain, Engelbert Mephu Nguifo

DASH introduces a dual-branch distillation framework to effectively compress class-conditional diffusion models by independently supervising both score branches, significantly preserving guidance fide…

View →
stat.MEcs.AIcs.LGRecentMay 27, 2026

Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors

Luyang Fang, Yongkai Chen, Jiazhang Cai, Ping Ma +1 more

The paper proposes Multi-Teacher Bayesian Knowledge Distillation (MT-BKD), a framework that uses Bayesian inference and teacher-informed priors to improve model compression, enhance predictive accurac…

View →
cs.LGcs.CLEmpiricalRecentJul 16, 2026

On-Policy Delta Distillation

Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han

This paper introduces a new distillation reward, the delta signal, for on-policy distillation in reinforcement learning, which captures changes induced by reasoning tuning and improves performance.

View →
cs.CVEmpiricalRecentJun 23, 2026

Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulation

Wuming Yang, Xiang Zhang, Hongmin Zhao

This paper proposes a framework called SPOFA to achieve stable knowledge distillation across different architectures.

View →
cs.AIRecentMay 29, 2026

Distilling LLM Feedback for Lean Theorem Proving

Gaetan Narozniak, Gérard Biau, Rémi Munos, Ahmad Rammal +1 more

The paper introduces Feedback Distillation, a novel training method that uses a language model's privileged feedback to provide token-level supervision, significantly improving complex reasoning tasks…

View →
cs.LGcs.AIRecentMay 29, 2026

Rethinking the Role of Temperature in Large Language Model Distillation

Hoang-Chau Luong, Lingwei Chen

This paper re-examines the role of temperature ($ au$) in LLM distillation, demonstrating that while Reverse KL (RKL) is often preferred, Forward KL (FKL) significantly outperforms RKL at higher tempe…

View →
cs.CVEmpiricalRecentJun 9, 2026

Mean Flow Distillation: Robust and Stable Distillation for Flow Matching Models

An Zhao, Shengyuan Zhang, Zhongjian Sun, Yixiang Zhou +4 more

This paper proposes a novel distillation framework for flow matching models that enables high-fidelity single-step generation.

View →
cs.AIcs.CRcs.CYMar 26, 2026

A Public Theory of Distillation Resistance via Constraint-Coupled Reasoning Architectures

Peng Wei, Wesley Shu

The paper proposes a theoretical framework, called constraint-coupled reasoning, to make AI models less susceptible to knowledge distillation by coupling high-level capabilities to internal stability…

View →
cs.LGcs.CRMay 12, 2026

Lossless Anti-Distillation Sampling

Zibo Diao, Jingchu Gai, Xinyue Ai, Zhang Zhang +2 more

The paper introduces Lossless Anti-Distillation Sampling (LADS), a novel sampling scheme that makes harvested data correlated for malicious distillers while ensuring benign users receive statistically…

View →
cs.LGEmpiricalRecentJun 16, 2026

Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?

Trisha Mittal, Akshay Mehra, Joshua Kimball

This paper evaluates the effectiveness of dataset distillation methods and finds that they are comparable to or worse than coreset selection methods on large-scale datasets.

View →
cs.LGRecentJun 1, 2026

Why Are DMD Students Lazy? Understanding the Copying Behavior in Few-Step Distillation

Shucheng Li, Iolo Jones, Alexander Tong, Michael M. Bronstein

This paper investigates the phenomenon of 'copying' in Distribution Matching Distillation (DMD), finding that high-dimensional distillation causes student models to spontaneously reproduce the teacher…

View →
cs.LGcs.AIRecentMay 27, 2026

Context Distillation as Latent Memory Management

Ziyang Zheng, Zeju Li, Xiangyu Wen, Jianyuan Zhong +4 more

The paper reframes context distillation as a latent memory management problem, proposing a modular framework using LoRA adapters and a Self-Gating mechanism for efficient, selective memory retrieval a…

View →
stat.MLcs.LGRecentJun 2, 2026

A Quantitative Approximation Framework for Flow Distillation in Diffusion Models

Weiguo Gao, Ming Li, Lei Shi, Hanfei Zhou

The paper develops a quantitative framework to analyze and improve flow distillation in diffusion models, providing stability guarantees and suggesting non-uniform time scheduling to reduce approximat…

View →
cs.LGcs.AIcs.SDRecentMay 30, 2026

Logit Distillation on Manifolds: Mapping by Learning

Yiru Yang, Junling Wang, Nishant Kumar Singh, Luohong Wu +1 more

The paper proposes a novel layer and point-wise projection mapping combined with LoRA injection to efficiently distill knowledge from a large teacher model to a small student model, significantly impr…

View →
eess.ASEmpiricalRecentJun 17, 2026

DASH: Dual-View Self-Distillation with Multi-Layer Hidden Representations for Robust Speech Recognition

Jaeeun Baik, Ui-Hyeop Shin, Jiwoon Lee, Woocheol Jeong +1 more

The paper proposes DASH, a self-distillation framework that improves ASR robustness in noisy environments by learning clean-noisy consistency from paired views.

View →
cs.LGcs.CLcs.DCEmpiricalRecentOct 13, 2026

DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss

Ruixuan Tu, Yuyang Dong

This paper presents DisaggKD, a knowledge distillation framework that separates teacher and student models across nodes to overcome per-GPU memory limits.

View →