Phase Matters: Characterizing Heterogeneous Vision-Language Inference on a Mobile SoC
This paper characterizes the performance of vision-language model inference on the Qualcomm SM8750 using FastVLM-0.5B as a case study, showing significant speedups and energy savings for different phases and encoders, and enabling previously unsupported encoders.
Provides detailed insights into the performance of vision-language model inference on mobile devices and enables unsupported encoders.
Before reading this…
Applications
- →Mobile vision-language model deployment
To understand this paper, make sure you know these concepts first:
- Basic understanding of machine learning models and mobile SoCsfind papers →
Abstract
More Like ThisRecent phone-class mobile SoCs expose practical NPU execution paths for on-device vision-language model (VLM) inference, but developers still lack phase-level guidance for mapping VLM pipelines across heterogeneous backends. We present a hardware-in-the-loop characterization of VLM inference on the Qualcomm SM8750 (Snapdragon 8 Elite), covering phase throughput, cache-state effects, 100-run thermal stability, energy, heterogeneous CPU/NPU pipeline configurations, and visual-token-budget sensitivity. Using FastVLM-0.5B as an end-to-end case study, together with encoder-only measurements across four architecture families, we show that phase matters: NPU execution is highly phase-dependent, delivering 1.64x speedup for prefill but only 1.18x for decode, while vision encoders achieve 20-45x speedups over CPU. These gains translate into 10.47 degrees C lower steady-state temperature and 2.52x lower energy, avoiding thermal throttling in always-on settings. Finally, we show that a four-step graph rewrite enables previously unsupported encoders, such as Phi-3.5-V, to reach the QNN path with up to 22x speedup, providing a practical porting recipe for mobile VLM deployment.