DepthWorld: 3D World Model for Robot Manipulation
This paper introduces a method to improve 3D world models for robotics by combining learned stereo depth with a joint factor graph and training a world model that jointly predicts multi-view RGB and depth.
The paper introduces a novel method for combining learned stereo depth with a joint factor graph to improve 3D world models for robotics.
Before reading this…
Applications
- →Policy evaluation
- →Improvement
- →Planning
To understand this paper, make sure you know these concepts first:
- Understanding of 3D world models and roboticsfind papers →
- Knowledge of computer vision and machine learningfind papers →
Abstract
More Like ThisWorld models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. We introduce a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving <0.7 px reprojection error on 90% of episodes for external cameras). We then train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained Variational Autoencoder (VAE) unchanged. Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.