Policy Gradient Steering: Interventions from Behavioral Objectives
This paper proposes Policy Gradient Steering (PGS), a new method for dynamically changing a large language model's behavior at inference time using reinforcement learning.
PGS formulates steering as a reinforcement learning problem, while existing methods do not.
Before reading this…
Applications
- →language models
- →gridworld environments
- →chess puzzles
- →competitive football
To understand this paper, make sure you know these concepts first:
- reinforcement learningfind papers →
- language modelsfind papers →
Abstract
More Like ThisActivation steering has emerged in large language models as a lightweight alternative for dynamically changing a model's behavior at inference time. However, we show that existing steering methods fail to steer even a simple policy in a two-route gridworld environment. To address this limitation, we propose Policy Gradient Steering (PGS), which formulates steering as a reinforcement learning problem. PGS accumulates gradients of a temporary behavioral objective over a small set of rollouts or demonstrations to construct a removable task vector. We first demonstrate the calibration and reversibility of PGS in a two-route gridworld environment. Using chess puzzles, we then evaluate independently fitted PGS vectors both in isolation and in combination, finding that compatible tactical objectives accumulate constructively. Finally, in competitive football, we show that PGS can alter specific team behaviors and that its effects transfer across opponents. Together, these results show that policy gradients provide a natural interface for constructing temporary and composable behavioral adaptations across diverse decision-making domains.