Summary

FantasyTalking is a cutting-edge framework for generating realistic talking portrait videos from a single static image and an audio track. Built upon a powerful video diffusion transformer architecture, FantasyTalking addresses the challenges of animating avatars with natural facial expressions, synchronized lip movements, and coherent body and background dynamics. The system employs a dual-stage audio-visual alignment strategy: first, it establishes global motion coherence across the entire scene by aligning audio-driven dynamics, and then it refines lip synchronization at the frame level using a specialized lip-tracing mask. This approach ensures that the generated video maintains both visual fidelity and precise synchronization with the provided audio, making it suitable for applications ranging from virtual avatars to animated storytelling and digital assistants.

Key Features

Dual-stage audio-visual alignment for coherent motion and precise lip synchronization
Facial-focused cross-attention module for identity preservation
Motion intensity modulation for controllable expressions and body movements
Supports various avatar styles, including realistic and cartoon
Animates close-up, half-body, and full-body poses
Open-source with available inference code and model weights

About FantasyTalking


A key innovation in FantasyTalking is its facial-focused cross-attention module, which replaces traditional reference networks to better preserve the identity of the subject throughout the animation. This module ensures that the unique facial features and expressions of the original portrait are retained, even as the model generates expressive and dynamic motion. Additionally, FantasyTalking integrates a motion intensity modulation network, allowing users to explicitly control the strength and style of facial expressions and body movements. This enables the generation of videos that are not only synchronized with speech but also rich in emotional nuance and natural movement, enhancing the realism and engagement of the animated avatars.


FantasyTalking stands out for its versatility and accessibility. It supports a wide range of avatar styles, from photorealistic to cartoon, and can animate characters and animals in various body poses, including close-up, half-body, and full-body orientations. The open-source release includes inference code and model weights, making it readily available for research, creative projects, and integration into broader video generation workflows. Extensive evaluations show that FantasyTalking outperforms previous methods in video quality, identity preservation, motion diversity, and lip synchronization, positioning it as a leading solution for realistic, controllable talking portrait generation.

Get more likes & reach the top of search results by adding this button on your site!

Embed button preview - Light theme
Embed button preview - Dark theme
TurboType Banner

Subscribe to the AI Search Newsletter

Get top updates in AI to your inbox every weekend. It's free!