✦
Summary
Uni3C is a unified 3D-enhanced framework from Alibaba DAMO Academy for precise and simultaneous control of both camera and human motion in video generation. Traditional approaches often treat camera movement and human animation as separate tasks, leading to limited flexibility and requiring extensive annotated datasets. Uni3C addresses these challenges with a plug-and-play control module, PCDController, which leverages unprojected 3D point clouds from monocular depth estimation to achieve fine-grained camera control, and it can be trained independently or in conjunction with foundational video diffusion models.
Key Features
Plug-and-play PCDController for precise 3D camera control using point clouds
Unified global 3D world guidance for consistent camera and human motion control
Uses unprojected 3D point clouds from monocular depth estimation
Aligns scenic point clouds and SMPL-X characters in a unified coordinate space
Supports complex motion transfer across different domains and video types
Built on the Wan2.1 video diffusion model
Modules can be trained independently, reducing dependence on jointly annotated data
Demonstrated superior performance on camera controllability and human motion quality
About Uni3C
A key innovation of Uni3C is its global 3D world guidance system, which aligns both scene geometry and human characters into a unified coordinate space during inference, enabling seamless, 3D-consistent generation of videos where camera trajectories and human motions are interdependent and can be controlled together. The system uses scenic point clouds for camera control and SMPL-X characters for human animation, bridging their relationship through 2D keypoints and rigid transformations, which allows complex motion transfer including scenarios where reference motions are sourced from different videos or domains.
Extensive benchmarking shows that Uni3C significantly outperforms existing methods in both camera controllability and human motion quality, and its modular design makes it compatible with various foundational video models, built on Wan2.1. While it excels in unified control, it does have limitations when human motions conflict with environmental constraints, potentially resulting in visual artifacts. It was published in ACM Transactions on Graphics (SIGGRAPH Asia 2025) with code released openly.