Sherpa: Teaching LLMs to Teach Adaptively
This paper introduces Sherpa, a multi-turn reinforcement learning framework that trains large language models to adapt their instruction to diverse students and become better aligned with human teachers.
This paper introduces a novel approach to training teacher LLMs to adapt to diverse students, which is different from existing methods that rely on demonstrations or predefined pedagogical criteria.
Before reading this…
Applications
- →AI tutoring
- →Education
To understand this paper, make sure you know these concepts first:
- Reinforcement learningfind papers →
- Large language modelsfind papers →
Abstract
More Like ThisLarge language models (LLMs) have become increasingly capable problem solvers, but being able to solve a problem is not the same as being able to teach it. Existing approaches to training LLMs as teachers rely on demonstrations, preference data, or predefined pedagogical criteria that specify what good teaching looks like. However, these signals are often not grounded in individual student learning outcomes, where effective teaching strategies can vary substantially across learners. To address this, we introduce Sherpa, a multi-turn reinforcement learning framework that instantiates multiple student archetypes with LLMs conditioned on distinct learning preferences and trains a teacher model to adapt its instruction by directly maximizing their learning outcomes. Teacher LLMs trained with Sherpa improve instructed students' performance across all archetypes by an average of 20.5 percentage points. Under MathTutorBench's evaluation, Sherpa raises the overall pedagogy score from 52.5% to 79.2%, indicating better teaching responses. Our human studies show that the trained teacher is preferred over the base model in 79.6% of pairwise comparisons. Together, Sherpa trains LLM teachers to adapt to diverse simulated students and become better aligned with human teachers, paving the road towards AI tutors teaching real students.