This paper proposes a new approach to multimodal reinforcement learning with verifiable rewards that preserves visually grounded guidance while suppressing inconsistent corrections.
The paper introduces a new method for validating privileged corrections against visual evidence, which is a novel approach compared to prior work.
Keywords
Before reading this…
Applications
To understand this paper, make sure you know these concepts first:
Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous token-level updates. To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence. We instantiate this paradigm with Visual-Attribution Gating (VAG), which uses the verifier-derived advantage as the optimization anchor and validates each privileged correction against the contrast between evidence-present and evidence-degraded teacher views. By admitting only corrections directionally supported by this visual contrast, VAG preserves visually grounded guidance while suppressing inconsistent residuals. Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.
Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
This paper proposes a new method, TPAE, to improve multimodal reasoning in large…
Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
The paper proposes a framework for post-training of reasoning language models us…
Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models
This paper proposes Steered-Teacher On-policy Self-Distillation (ST-OPSD) to add…
Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unificatio…
This paper proposes a unified autoregressive framework (UniAR) for integrating v…
A Lightweight Dual-Branch Visual Front-End for Efficient Audio-Visual Target Speaker Extraction
This paper proposes a dual-branch visual front-end for Audio-Visual Target Speak…
VISTA: A Visual Harness for Reasoning in an Interactive World
The authors introduce VISTA, a visual harness that enables multimodal models to…
Semantic-Critical Regions for Video-to-Audio Generation
This paper introduces a video-to-audio generation framework that uses fine-grain…
On the Position Bias of On-Policy Distillation
The paper improves the learning efficiency of reinforcement learning through a n…