Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Models
This paper investigates the dialectal robustness of large language models and speech language models using Japanese dialects.
This paper is novel in its investigation of dialectal robustness in speech language models and its comparison with text-based models.
Before reading this…
Applications
- →Speech recognition
- →Spoken language processing
To understand this paper, make sure you know these concepts first:
- Knowledge of language models and speech processingfind papers →
- Understanding of dialectal variationfind papers →
Abstract
More Like ThisDialogue systems based on large language models (LLMs) have advanced significantly in recent years. However, dialectal variation remains a major challenge, particularly for systems that process spoken input. LLM-based speech language models (SLMs), which integrate LLMs with speech processing components, show promise for spoken language tasks, yet their ability to comprehend dialects has not been sufficiently studied. Moreover, it remains unclear how the dialectal understanding of the base LLM affects SLM performance. This study investigates the dialectal robustness of both LLMs and SLMs using Japanese dialects as a test case. We define robustness as the ratio of performance on dialectal versus standard inputs, enabling fair comparisons. Our experiments show that SLM robustness correlates with that of their text-based counterparts. Furthermore, training with dialectal data and fine-tuning the speech encoder each improves robustness in SLMs.