Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition
This paper introduces EmoSpeechBrain, a multimodal framework for emotion recognition that suppresses EEG noise and improves accuracy.
This paper introduces a new method for suppressing EEG noise and improving the accuracy of multimodal emotion recognition.
Before reading this…
Applications
- →Emotion recognition in human-computer interaction
- →Affective computing
To understand this paper, make sure you know these concepts first:
- Basic knowledge of machine learning and neural networksfind papers →
- Understanding of EEG and speech modalitiesfind papers →
Abstract
More Like ThisMultimodal emotion recognition (MER) increasingly pairs EEG with speech, treating internal neural signals and external vocal expression as informative views of affect. In practice, naive fusion underperforms the stronger single modality, because EEG artifacts inject noise that corrupts the shared representation. We introduce EmoSpeechBrain, a multimodal framework built on the insight that noise suppression is a precondition for effective fusion. Its EEG encoder uses differential attention, taking the difference between two attention maps to cancel shared noise and isolate discriminative neural activity. An attention-based gating adapter aligns both modalities in a shared space and weights each one's contribution to the prediction. On two datasets - PME4 and EAV, EmoSpeechBrain improves MER accuracy by up to 12.9% over other state-of-the-art (SOTA) EEG encoders, and surpasses unimodal speech and EEG baselines by up to 13.1% and 23.1%. These results show that once EEG noise is suppressed, fusion delivers gains that naive combination cannot.