On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning
This paper improves test-time reinforcement learning by reducing errors and adapting to unlabeled target-domain data.
The paper proposes a novel method, PC-Dr.DPO, to condition within-prompt robust reweighting on observed vote reliability.
Before reading this…
Applications
- →language understanding
- →knowledge-intensive reasoning
To understand this paper, make sure you know these concepts first:
- test-time reinforcement learningfind papers →
- language modelsfind papers →
Abstract
More Like ThisTest-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose \emph{Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO)}, which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. We further introduce a cache-based mechanism to reduce self-supervised forgetting during online adaptation. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.