IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages
The paper introduces IndicContextEval, a multilingual benchmark to evaluate the use of contextual signals in AudioLLMs.
The paper introduces a novel benchmark and evaluation framework for assessing the use of contextual signals in AudioLLMs.
Before reading this…
Applications
- →improving speech recognition models
To understand this paper, make sure you know these concepts first:
- speech recognitionfind papers →
- contextual signalsfind papers →
Abstract
More Like ThisAudioLLMs enable speech recognition conditioned on textual prompts such as domain descriptions or entity lists. However, it remains unclear whether these models genuinely utilise such context or rely on parametric knowledge learned during pretraining. Existing benchmarks cannot answer this question because they evaluate transcription under fixed prompting conditions and rarely include explicit contextual inputs. We introduce IndicContextEval, a 56-hour multilingual benchmark of natural speech from 555 speakers across 8 Indian languages and 23 professional domains. We design a 7-level prompting framework that progressively introduces contextual signals, including metadata, natural-language descriptions, entity lists in English and native script, and adversarial prompts with incorrect entities. Evaluating five models reveals substantial differences in context utilisation behaviour, highlighting the need for explicit evaluation of contextual grounding in AudioLLMs.