✦
Summary
OmniTalker is a real-time talking head generation framework from Alibaba's Tongyi Lab that produces synchronized speech and video directly from text, all while replicating the speech and facial style of a reference video. Unlike traditional cascaded systems that combine separate text-to-speech and audio-driven talking head models, OmniTalker employs a unified, end-to-end approach, eliminating common issues such as audio-visual asynchrony and stylistic mismatches and resulting in highly natural, expressive and lifelike talking head videos. It is the first one-shot framework capable of jointly modeling speech and facial styles in real time.
Key Features
Unified end-to-end framework for real-time text-to-video talking head generation
Dual-branch diffusion transformer for synchronized audio and visual output
Zero-shot style replication from a single reference video
Emotionally expressive video generation with support for multiple emotions
Cross-lingual support for English and Chinese with style preservation
Audio-visual fusion module ensuring temporal synchronization and stylistic coherence
In-context reference learning with a masked-infilling strategy, no separate style extractor needed
Real-time inference at 25 FPS on a single RTX 4090 GPU
About OmniTalker
At the core of OmniTalker is a dual-branch diffusion transformer architecture: the audio branch synthesizes high-quality speech from text, while the visual branch predicts detailed head poses and facial dynamics, with cross-modal fusion modules integrating information between the two modalities at shallow layers to ensure perfect temporal synchronization and stylistic coherence. The in-context reference learning module, using a masked-infilling strategy that leverages the diffusion transformer's in-context learning capability, extracts both speech and facial style characteristics from a single reference video, allowing seamless zero-shot replication without dedicated style extraction modules or large datasets.
OmniTalker is engineered for efficiency and accessibility, delivering real-time performance at 25 frames per second on a single NVIDIA RTX 4090 GPU without compromising output quality. The platform supports both English and Chinese, with cross-lingual generation capabilities that preserve the speaker's style even when switching languages, and it can generate emotionally expressive videos across a range of emotions. Presented at NeurIPS 2025, it is suitable for content creators, educators, marketers and anyone seeking personalized, style-consistent talking head videos, including virtual presenters and interactive video chat.