Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models
This paper proposes Steered-Teacher On-policy Self-Distillation (ST-OPSD) to address failure modes in on-policy self-distillation.
This paper introduces a novel method, Steered-Teacher OPSD, to address the failure modes of on-policy self-distillation, which is not addressed by prior work.
Keywords
Before reading this…
Applications
- →Improving the accuracy of on-policy self-distillation
- →Restoring reconsideration and reference-hallucination rates to base-model levels
To understand this paper, make sure you know these concepts first:
- On-policy self-distillationfind papers →
- Activation steeringfind papers →
Abstract
More Like ThisOn-policy self-distillation (OPSD) trains a student on its own reasoning trajectories using token-level supervision from a reference-conditioned teacher initialized from the same model. However, access to the reference can induce teacher behaviors that undermine independent reasoning in thinking models, leading to reference hallucinations, where the student cites an unavailable reference to justify reasoning steps, and reconsideration suppression, where the teacher discourages verification and revision at uncertain steps. We introduce Steered-Teacher OPSD (ST-OPSD) to jointly address these failure modes through activation steering. ST-OPSD extracts two directions from model-generated reasoning trajectories and applies them to the teacher's hidden states during distillation target computation, suppressing reference-invoking behavior and encouraging reconsideration with marker-family calibration. The student learns from the resulting teacher distributions without being steered. Experiments on AIME and HMMT show that ST-OPSD improves accuracy over vanilla OPSD and other interventions while restoring reconsideration and reference-hallucination rates to base-model levels.