SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs
This paper introduces SPOT-E, a plug-and-play test-time method that improves the performance of vision-language models on evidence-intensive tasks by producing question-conditioned spotlights.
This paper introduces a novel approach to improving the performance of VLMs on evidence-intensive tasks by producing question-conditioned spotlights.
Before reading this…
Applications
- →Improving the performance of VLMs on evidence-intensive tasks
To understand this paper, make sure you know these concepts first:
- Vision-language modelsfind papers →
- Evidence-intensive tasksfind papers →
Abstract
More Like ThisVision-language models (VLMs) often underperform on evidence intensive tasks because decisive visual evidence are small, localized, and easy to overlook, leading to failures in evidence readout even when high-level reasoning is intact. Prior inference-time visual interventions can improve grounding without retraining, but they are largely open-loop and lack a mechanism to verify whether highlighted evidence is actually used. We study answer-span prediction entropy as a model-internal feedback signal and show that naive entropy minimization is ambiguous, since low entropy may arise from evidence-grounded confidence or shortcut collapse. To resolve this ambiguity, we introduce low-entropy anchors and an entropy-shaping objective that reduces answer uncertainty while preserving baseline high-confidence tokens. We instantiate this principle in SPOT-E, a plug-and-play test-time method that produces question-conditioned spotlights, optimized per instance via light-weight tuning based on Group Relative Policy Optimization (GRPO). Across all benchmarks and different VLM families, SPOT-E yields consistent gains and improved robustness under visual corruptions. Code is publicly available at: \url{https://github.com/YinBo0927/SPOT-E}