Towards Robust Generative Speech Enhancement Using Vector Quantisation-Based Neural Audio Codec
This paper investigates modelling strategies in continuous and discrete latent spaces in VQ-based neural audio codec speech enhancement.
This paper introduces a new approach to speech enhancement using VQ-based neural audio codec and continuous and discrete latent space modelling.
Before reading this…
Applications
- →Speech enhancement
- →Audio processing
To understand this paper, make sure you know these concepts first:
- Neural networksfind papers →
- Vector quantisationfind papers →
Abstract
More Like ThisThis work investigates modelling strategies in continuous and discrete latent spaces in the vector quantisation (VQ)-based neural audio codec (NAC) speech enhancement (SE), along with the role of VQ regularisation. We propose cNAC-SE and dNAC-SE frameworks that predict continuous representations and discrete tokens in latent space, respectively. Theoretical analysis and visualisations in latent space are performed to exhibit their inherent modelling mechanisms. Experimental results show that the fully fine-tuned cNAC-SE model consistently outperforms all dNAC-SE variants across diverse test conditions and achieves leading performance among established generative approaches in DNS-MOS metrics. Comparison with the discriminative counterpart shows that VQ enhances robustness through an intrinsic effect of clean-prior-constrained regularisation, independent of discrete token processing. This highlights the transferable value of VQ regularisation to other continuous modelling methods.