PoEM: Predicting RL Outcomes from Existing Policies
This paper introduces PoEM, a framework to predict the outputs of reinforcement learning on a new reward function using pre-trained models.
This paper introduces a novel approach to predicting RL outcomes without running RL on a new reward function, which is different from prior work that requires running RL from scratch.
Before reading this…
Applications
- →Text and image modalities
- →Synthetic and real rewards
To understand this paper, make sure you know these concepts first:
- Reinforcement learningfind papers →
- Post-trainingfind papers →
Abstract
More Like ThisFoundation models are post-trained with reinforcement learning (RL) to maximize specific rewards, such as human alignment, correctness, or instruction following. This post-training process is computationally intensive, sometimes unstable, and has to be run from scratch every time the reward model changes or when we want to combine multiple rewards. We hence ask: given a new reward function, is it possible to predict the RL outcomes without actually running RL on it? We answer this in the affirmative by introducing PoEM, a framework to predict the outputs of RL on a new reward function using a set of models already post-trained on other rewards. First, we show that if the new reward function can be written as a linear combination of existing ones, then the new policy in log-space can be written as a linear combination of the existing log-policies. Surprisingly, even in cases where the rewards are not linearly connected, we observe that often log-policies from RL training span an approximately low-rank subspace across rewards. To our benefit, the weighting coefficients for this combination can be estimated using only the reward or basis policy outputs on the samples. We turn these observations into an algorithm that takes post-trained models and a new reward function, and approximates the target RL policy without actually running any additional RL training. We experimentally validate our approach across synthetic and real rewards, spanning both text and image modalities.