The paper proposes SpeechKV, a method to compress speech sequences inside large language models using a learned pooling, maintaining performance and delivering decoding speedup.
SpeechKV is novel as it applies a learned pooling to the KV cache of speech tokens inside the LLM, unlike previous methods that compress speech at the adapter level.
Before reading this…
Applications
- →Speech recognition, Text-to-speech, Language modeling.
To understand this paper, make sure you know these concepts first:
- Understanding of large language models, Speech recognition, Text-to-speech.find papers →
Abstract
More Like ThisSpeech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length.