From Tokens to Faces: Investigating Discrete Speech Representations for 3D Facial Animation
This paper evaluates four speech representation families for 3D facial synthesis and introduces an Audio Visual Text-to-Speech pipeline.
The paper introduces a novel Audio Visual Text-to-Speech pipeline that leverages discrete representations for speech and 3D facial motion.
Before reading this…
Applications
- →3D facial animation
- →text-to-speech systems
To understand this paper, make sure you know these concepts first:
- speech processingfind papers →
- 3D computer visionfind papers →
Abstract
More Like ThisThe choice of speech representation is critical in speech-driven 3D facial animation. Representations differ in what they encode: SSL features emphasize segmental and semantic cues, neural codecs yield latents optimized for acoustic reconstruction, and ASR-style objectives produce label-based spaces. We evaluate four speech representation families for 3D facial synthesis, comparing their facial reconstruction quality across two facial decoders using objective metrics and a perceptual evaluation. We additionally conduct probing analyses that relate tokenized representations to phonetic units and to articulatory deformations. We found that encoding phonetic classes is beneficial for accurate facial animation prediction on both semantic and label-based representations with comparable facial animation quality. From the latter, we introduce an Audio Visual Text-to-Speech (AVTTS) pipeline that leverages, as a shared space, discrete representations to decode speech and 3D facial motion.