Summary

RealisDance-DiT is an open-source model for controllable character animation in diverse, real-world scenarios, built upon the Wan-2.1 video foundation model. Its core insight is that, as long as the foundation model is powerful enough, straightforward model modifications with flexible fine-tuning strategies can largely address the challenges of character animation, rather than elaborate bypass networks. The model's minimal yet effective modifications, combined with advanced fine-tuning strategies such as low-noise warmup and large batches with small iterations, enable highly realistic and physically consistent animation of characters based on pose and appearance guidance.

Key Features

Highly controllable character animation with support for rare poses and stylized characters
Handles complex lighting, dynamic scenes and character-object interactions with physical realism
Minimal architectural modifications for efficient fine-tuning and strong generalization
Integrates three pose conditions (HaMeR, DWPose, SMPL-CS) and reference images for precise motion guidance
Low-noise warmup and large-batch fine-tuning strategies preserving the foundation model's priors
Superior performance on TikTok, UBC fashion and the new RealisDance-Val benchmarks
Shows potential for multi-character control despite single-character training
Open-source and extensible for research and production use

About RealisDance-DiT


A notable strength of RealisDance-DiT is its ability to generalize across a wide range of animation challenges, handling rare poses, stylized characters, complex lighting, dynamic scenes and intricate character-object interactions while maintaining high fidelity and temporal consistency. The model employs three pose conditions, HaMeR, DWPose and SMPL-CS, encoded alongside a reference image using the original Wan-2.1 VAE, and through an improved spatially shifted Rotary Position Embedding (RoPE) it achieves superior alignment between character motion and environmental factors like lighting and object manipulation, retaining object continuity and generating physically plausible movements.


RealisDance-DiT's performance has been validated on multiple benchmarks, including the TikTok dataset, the UBC fashion video dataset and the newly introduced RealisDance-Val dataset of 100 videos capturing real-world animation challenges, where it consistently ranks first or second in key metrics such as FVD and FID, outperforming leading methods such as Animate-X, ControlNeXt and commercial products like ViggleAI. Its open-source nature and robust generalization make it an invaluable tool for researchers, animators and developers creating high-quality, controllable character animations for entertainment, virtual production or research.

Get more likes & reach the top of search results by adding this button on your site!

Embed button preview - Light theme
Embed button preview - Dark theme
TurboType Banner

Subscribe to the AI Search Newsletter

Get top updates in AI to your inbox every weekend. It's free!