cs.AIcs.CVEmpirical

SceneActBench: Can Agents Act on the 3D Scenes They See?

Yifei Zhao, Xiangxin Zhou, Wenhao Yang, Jiaqi Tang, Pu Jian, Huanjin Yao, Jiarui Yao, Haowei Lin, Chunchao Guo, Zhuo Chen, Wenkai Lyu, Jianzhu Ma, Xueqian Wang, Wenxi Zhu

Jul 24, 2026

PDF HTML

AI Summaryllama-3.1-8b-instruct

The paper introduces SceneActBench, a benchmark for evaluating vision-language model agents' ability to perform actions on multi-object 3D scenes.

Introduced SceneActBench, a new benchmark for evaluating agents' ability to act on 3D scenes based on visual input.

Keywords

vision-language model, 3D scenes, benchmark, evaluation, multi-object

Before reading this…

Understanding of vision-language models, 3D scenes, benchmarking

Applications

→AI model evaluation, 3D scene understanding

Skill Ladder

To understand this paper, make sure you know these concepts first:

Understanding of vision-language models, 3D scenes, benchmarkingfind papers →

Abstract

More Like This

Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated. We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6-50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.