GIVE: Grounding Human Gestures in Vision-Language-Action Models
This paper proposes GIVE, an approach that enhances VLA models with human gesture understanding to improve Human-Robot Interaction.
This paper introduces a novel approach to incorporating gesture information into VLA models without architectural modifications.
Before reading this…
Applications
- →Human-Robot Interaction
- →Robotics
To understand this paper, make sure you know these concepts first:
- Vision-Language-Action modelsfind papers →
- Human-Robot Interactionfind papers →
Abstract
More Like ThisHuman communication is inherently multimodal, where language is often accompanied by non-verbal cues such as gestures to convey intentions. However, current Vision-Language-Action (VLA) models treat robotic manipulation as a pure text-driven task, overlooking the important role of gestures in Human-Robot Interaction (HRI). This often leads to inaccurate intent grounding and unreliable manipulation when language instructions are ambiguous or underspecified. To address this challenge, we propose GIVE (Gesture Intent via Visual-Semantic Enhancement), an effective approach that enhances pre-trained VLA models with human gesture understanding without architectural modifications. Specifically, GIVE incorporates gesture information through two complementary pathways: a visual pathway that overlays hand skeletons and fingertip rays onto robot observations for explicit object grounding, and a semantic pathway that generates high-level descriptions of human gestures and task instructions for robust intent grounding. By jointly leveraging visual and semantic guidance, GIVE enables VLA policies to better associate gestures with manipulation behaviors and adapt to dynamic interaction intents. In real-world HRI experiments, GIVE substantially outperforms the baseline, improving target object recognition accuracy by 40% and overall task success rate by 80%, while demonstrating strong robustness and generalization to unseen spatial layouts and diverse participants.