✦
Summary
RealisDance-DiT is an open-source model for controllable character animation in diverse, real-world scenarios, built upon the Wan-2.1 video foundation model. Its core insight is that, as long as the foundation model is powerful enough, straightforward model modifications with flexible fine-tuning strategies can largely address the challenges of character animation, rather than elaborate bypass networks. The model's minimal yet effective modifications, combined with advanced fine-tuning strategies such as low-noise warmup and large batches with small iterations, enable highly realistic and physically consistent animation of characters based on pose and appearance guidance.
Key Features
Highly controllable character animation with support for rare poses and stylized characters
Handles complex lighting, dynamic scenes and character-object interactions with physical realism
Minimal architectural modifications for efficient fine-tuning and strong generalization
Integrates three pose conditions (HaMeR, DWPose, SMPL-CS) and reference images for precise motion guidance
Low-noise warmup and large-batch fine-tuning strategies preserving the foundation model's priors
Superior performance on TikTok, UBC fashion and the new RealisDance-Val benchmarks
Shows potential for multi-character control despite single-character training
Open-source and extensible for research and production use
About RealisDance-DiT
A notable strength of RealisDance-DiT is its ability to generalize across a wide range of animation challenges, handling rare poses, stylized characters, complex lighting, dynamic scenes and intricate character-object interactions while maintaining high fidelity and temporal consistency. The model employs three pose conditions, HaMeR, DWPose and SMPL-CS, encoded alongside a reference image using the original Wan-2.1 VAE, and through an improved spatially shifted Rotary Position Embedding (RoPE) it achieves superior alignment between character motion and environmental factors like lighting and object manipulation, retaining object continuity and generating physically plausible movements.
RealisDance-DiT's performance has been validated on multiple benchmarks, including the TikTok dataset, the UBC fashion video dataset and the newly introduced RealisDance-Val dataset of 100 videos capturing real-world animation challenges, where it consistently ranks first or second in key metrics such as FVD and FID, outperforming leading methods such as Animate-X, ControlNeXt and commercial products like ViggleAI. Its open-source nature and robust generalization make it an invaluable tool for researchers, animators and developers creating high-quality, controllable character animations for entertainment, virtual production or research.