EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games
The paper introduces EMAgnet, a new regularization target for policy gradient methods that adapts to the agent's improving strategy.
The paper introduces a novel regularization target that adapts to the agent's improving strategy, unlike existing methods that use a fixed regularization target.
Before reading this…
Applications
- →Game playing
- →Reinforcement learning
To understand this paper, make sure you know these concepts first:
- Policy gradient methodsfind papers →
- Self-playfind papers →
Abstract
More Like ThisRecent work has established that regularized policy gradient methods such as PPO, when used in self-play, can match or exceed specialized game-theoretic algorithms for solving two-player zero-sum imperfect-information games. The uniform distribution has emerged as a strong policy regularization target for this purpose, but it regularizes equally toward all actions regardless of their viability. We introduce EMAgnet, which instead regularizes toward an exponential moving average (EMA) of the last-iterate policy's parameters, providing an adaptive regularization target that evolves with the agent's improving strategy. We evaluate EMAgnet on both standard two-player zero-sum benchmarks and modified benchmarks with exploration challenges and large numbers of strictly dominated strategies. Relative to PPO self-play with uniform-magnet regularization under both linear and power-law annealing schedules, EMAgnet achieves lower exploitability in the majority of tested environments, with consistent performance gains across games containing strictly dominated strategies.