Through the PRISM: Preference Representation in Intermediate States of Video Diffusion Models
The paper introduces PRISM, a method for decoding preference signals from noisy latents using a lightweight Query-based Aggregation head and a frozen video diffusion backbone, achieving state-of-the-art preference accuracy and noise-robustness.
The authors propose a new approach for evaluating video generation models by decoding preference signals directly from noisy latents, which enables early-stage filtering of suboptimal candidates and reduces computation.
Before reading this…
Applications
- →Video generation
- →Preference learning
To understand this paper, make sure you know these concepts first:
- Understanding of video generation modelsfind papers →
- Familiarity with diffusion modelsfind papers →
Abstract
More Like ThisEvaluating video generation with clean, pixel-based reward models disconnects evaluation from the noisy diffusion process and incurs massive VAE decoding costs. In this paper, we challenge this paradigm by asking a fundamental question: Can a powerful video generator inherently discriminate preferences directly from noisy latents? To answer this, we introduce \textbf{PRISM} (\textbf{P}reference \textbf{R}epresentation in \textbf{I}ntermediate \textbf{S}tates of Diffusion \textbf{M}odels). PRISM employs a lightweight Query-based Aggregation head with a frozen video diffusion backbone to decode preference signals from noisy latents. Surprisingly, PRISM not only achieves SOTA preference accuracy but also unlocks strong noise-robustness, which enables early-stage Best-of-$N$ sampling. This allows for filtering suboptimal candidates at the very beginning of denoising, drastically reducing computation while boosting video quality. We also reveal a strong positive correlation between a backbone's generative performance and its inherent evaluative power, enabling self-improving video backbones.