The paper studies the design of context for self-distillation in language models, finding that step-aligned feedback yields the largest gains.

The paper proposes Multi-Teacher Bayesian Knowledge Distillation (MT-BKD), a framework that uses Bayesian inference and teacher-informed priors to improve model compression, enhance predictive accurac…

The paper introduces Feedback Distillation, a novel training method that uses a language model's privileged feedback to provide token-level supervision, significantly improving complex reasoning tasks…

This paper proposes a framework for efficient fire classification using a hybrid uncertainty-aware multi-stage progressive knowledge distillation method.

The paper proposes a theoretical framework, called constraint-coupled reasoning, to make AI models less susceptible to knowledge distillation by coupling high-level capabilities to internal stability…

This paper introduces a 'Sleep' paradigm for machine learning models to continually learn and transfer knowledge.

This paper proposes a novel method for training acceleration of speech foundation model distillation to speed up model deployment.

The paper proposes Distribution-Aligned Self-Distillation (DASD) to improve self-distillation by dynamically filtering high-perplexity tokens, thereby preserving useful logical knowledge while suppres…

This paper proposes SimOPD, a method for on-policy distillation that transfers teacher internal knowledge through response similarities.

COLLEAGUE.SKILL introduces an automated system that distills heterogeneous traces of human expertise and role-specific knowledge into portable, inspectable, and usable AI skill packages.

This paper presents DisaggKD, a knowledge distillation framework that separates teacher and student models across nodes to overcome per-GPU memory limits.

LoopFM proposes a novel framework to significantly improve knowledge distillation for recommendation systems by structuring the rich intermediate embeddings of large foundation models as input feature…

The paper introduces a systematic framework to convert large Mixture-of-Experts (MoE) models into memory-efficient, fully dense architectures, achieving superior performance compared to traditional pr…

The paper reframes context distillation as a latent memory management problem, proposing a modular framework using LoRA adapters and a Self-Gating mechanism for efficient, selective memory retrieval a…

This paper proposes an HPC-aware methodology for Knowledge Distillation (KD) that decouples teacher and student partitioning efficiently, achieving up to 67% higher samples-per-second than the widely…

This paper evaluates the effectiveness of dataset distillation methods and finds that they are comparable to or worse than coreset selection methods on large-scale datasets.