Transcript-Free Flow-Matching Text-to-Speech via Speech Feature Conditioning
The paper proposes RTFree-F5, a method to make flow-matching TTS models like F5-TTS independent of reference transcripts, improving performance and naturalness for dysarthric speakers.
Proposing a new method, RTFree-F5, to replace reference transcripts with self-supervised speech representations in text-to-speech models
Before reading this…
Applications
- →Assistive technologies for people with speech disorders
To understand this paper, make sure you know these concepts first:
- Understanding of text-to-speech modelsfind papers →
- Familiarity with self-supervised learningfind papers →
Abstract
More Like ThisRecent flow-matching text-to-speech (TTS) models, such as F5-TTS, rely on a reference transcript at inference time, obtained from an external ASR system. This dependency makes zero-shot TTS brittle for accented or dysarthric speakers, precisely the scenarios where it is most needed. Moreover, we find that text-based reference conditioning can propagate atypical acoustic patterns from atypical speech into synthesis, even when ground-truth transcripts are available. To address this, we propose RTFree-F5, which replaces the reference transcript with continuous self-supervised speech representations mapped into F5-TTS's text-conditioning space via a lightweight adapter, while reusing the pretrained checkpoint. On dysarthric speech, RTFree-F5 reduces WER from 24.6% to 10.4%, surpassing even the ground-truth reference transcript baselines, while improving naturalness and remaining competitive on standard benchmarks without requiring any reference transcript.