Speech Encoder Fusion for LLM-based Automatic Speech Recognition
This paper investigates whether combining multiple pre-trained speech encoders can improve speech-aware large language models for automatic speech recognition.
This paper introduces a novel approach to combining multiple pre-trained speech encoders to improve speech-aware language models.
Before reading this…
Applications
- →Automatic speech recognition
- →Speech recognition
To understand this paper, make sure you know these concepts first:
- Pre-trained speech encodersfind papers →
- Large language modelsfind papers →
Abstract
More Like ThisSpeech-aware large language models (LLMs) can incorporate speech through pre-trained acoustic encoders that project speech features into the LLM embedding space. While the choice of the speech encoder critically influences performance, different encoders often exhibit complementary strengths, motivating their combination. In this work, we investigate whether fusing multiple pre-trained speech encoders can enhance speech-aware LLMs for automatic speech recognition (ASR). We explore several fusion strategies beyond simple feature concatenation, including learned combinations and Transformer-based fusion architectures, and evaluate them across mono- and multilingual ASR settings as well as diarized speech recognition. Our results indicate that carefully fusing multiple parallel speech encoders improves downstream performance in all scenarios with limited computational overhead.