Real-Time Voice AI Hears but Does Not Listen
The paper evaluates four leading production voice systems and finds that they often ignore the emotional tone and delivery of speech, leading to incorrect decisions.
The paper highlights a previously unknown disconnect between the perception and action of voice AI systems.
Before reading this…
Applications
- →Improving voice AI systems to better understand emotional tone and delivery
To understand this paper, make sure you know these concepts first:
- Understanding of voice AI systemsfind papers →
- Knowledge of speech recognitionfind papers →
Abstract
More Like ThisSpeech conveys information through both words and vocal delivery. We evaluate four leading production realtime voice systems-OpenAI's GPT Realtime 2, Google's Gemini 3.1 Flash Live, and Alibaba's Qwen3.5 Omni Plus and Omni Flash-on tasks where the words and the delivery patterns both convey meaningful information. Across three consequential scenarios, all four systems act on the words rather than the voice. They end calls with crying callers who insist nothing is wrong, approve wire transfers authorized in frightened voices, and enroll callers whose agreement is clearly sarcastic. Surprisingly, this is often not a failure of perception. When asked directly, three of the four systems reliably identify the distress, fear, or sarcasm they later ignore when making decisions. We observe a similar pattern when these realtime voice systems estimate accent and age, as their responses frequently follow the biases of the words rather than the acoustic properties of the speaker. We term this disconnect between perception and action the emotional intelligence gap of voice AI. Prompting systems to explicitly attend to vocal delivery improves performance only partially and inconsistently. Our findings show that current realtime voice AI systems often behave as if speech had been reduced to a transcript, suggesting that they should be used with caution in settings where the tone and emotion of delivery convey important information.