VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control
Sixiao Zheng, Minghao Yin, Wenbo Hu, Xiaoyu Li, Ying Shan, Yanwei Fu
2026-01-09
Summary
This paper introduces VerseCrafter, a new system for creating realistic videos of dynamic scenes. It focuses on giving users precise control over what happens in the video, including both the movement of objects and the camera's viewpoint.
What's the problem?
Existing methods for creating video simulations struggle with accurately controlling both the camera and multiple objects moving around in a scene. Videos are usually based on what we *see* (2D images), making it hard to understand and manipulate the actual 3D world happening within them. Also, getting enough data to train these systems is difficult because it requires detailed information about how things move in 3D over time.
What's the solution?
VerseCrafter solves this by creating a 4D representation of the scene – think of it as a 3D world that changes over time. It uses a combination of a static background and tracks objects as 3D shapes that evolve over time, capturing their possible locations. Users can then specify how they want objects and the camera to move, and VerseCrafter generates a video that perfectly matches those instructions. To overcome the data shortage, the researchers developed a way to automatically extract the necessary movement information from existing videos.
Why it matters?
This work is important because it allows for much more precise and intuitive control over video creation. Instead of struggling to get a video to look a certain way, users can directly specify the actions and camera angles they want, leading to more realistic and customizable video content. This could be useful for creating special effects, simulations, or even virtual reality experiences.
Abstract
Video world models aim to simulate dynamic, real-world environments, yet existing methods struggle to provide unified and precise control over camera and multi-object motion, as videos inherently operate dynamics in the projected 2D image plane. To bridge this gap, we introduce VerseCrafter, a 4D-aware video world model that enables explicit and coherent control over both camera and object dynamics within a unified 4D geometric world state. Our approach is centered on a novel 4D Geometric Control representation, which encodes the world state through a static background point cloud and per-object 3D Gaussian trajectories. This representation captures not only an object's path but also its probabilistic 3D occupancy over time, offering a flexible, category-agnostic alternative to rigid bounding boxes or parametric models. These 4D controls are rendered into conditioning signals for a pretrained video diffusion model, enabling the generation of high-fidelity, view-consistent videos that precisely adhere to the specified dynamics. Unfortunately, another major challenge lies in the scarcity of large-scale training data with explicit 4D annotations. We address this by developing an automatic data engine that extracts the required 4D controls from in-the-wild videos, allowing us to train our model on a massive and diverse dataset.