FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis
Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, Mu Xu
2025-04-10
Summary
This paper talks about FantasyTalking, a tool that creates realistic talking videos from a single photo by making the character’s mouth, face, and body move naturally with sound.
What's the problem?
Current methods can’t make characters from photos look natural when talking—they mess up facial details, body movements, or backgrounds, making videos look stiff or fake.
What's the solution?
FantasyTalking uses a two-step process: first, it syncs the whole scene (body, background) to the audio, then fine-tunes lip movements frame by frame. It keeps the face looking like the original photo while letting other parts move freely.
Why it matters?
This helps create lifelike avatars for movies, games, or virtual assistants that look and move naturally, making digital characters feel more real and expressive.
Abstract
Creating a realistic animatable avatar from a single static portrait remains challenging. Existing approaches often struggle to capture subtle facial expressions, the associated global body movements, and the dynamic background. To address these limitations, we propose a novel framework that leverages a pretrained video diffusion transformer model to generate high-fidelity, coherent talking portraits with controllable motion dynamics. At the core of our work is a dual-stage audio-visual alignment strategy. In the first stage, we employ a clip-level training scheme to establish coherent global motion by aligning audio-driven dynamics across the entire scene, including the reference portrait, contextual objects, and background. In the second stage, we refine lip movements at the frame level using a lip-tracing mask, ensuring precise synchronization with audio signals. To preserve identity without compromising motion flexibility, we replace the commonly used reference network with a facial-focused cross-attention module that effectively maintains facial consistency throughout the video. Furthermore, we integrate a motion intensity modulation module that explicitly controls expression and body motion intensity, enabling controllable manipulation of portrait movements beyond mere lip motion. Extensive experimental results show that our proposed approach achieves higher quality with better realism, coherence, motion intensity, and identity preservation. Ours project page: https://fantasy-amap.github.io/fantasy-talking/.