At the core of TaoAvatar is an advanced 3D Gaussian Splatting (3DGS) technology, which enables the creation of avatars with rich details and natural movements while maintaining low computational overhead. The pipeline begins with a personalized, clothed human template, and then uses a StyleUnet-based network to capture complex, pose-dependent deformations and high-frequency appearance details. These are distilled into a lightweight MLP-based model, making the avatars efficient enough for real-time rendering at up to 90 frames per second, even on mobile hardware. The system also incorporates blend shapes to enhance fine-grained control over expressions and body motion, ensuring avatars respond naturally to audio and visual cues.
TaoAvatar stands out for its flexibility and extensibility, supporting interactive demos and integration with speech and gesture control pipelines. The avatars are dynamically driven by models that synchronize speech, facial expressions, and gestures, resulting in digital humans that appear emotionally expressive and responsive. The framework is supported by the TalkBody4D dataset, which provides high-resolution multi-view sequences for training and evaluation. With its robust performance, low resource requirements, and ability to deliver lifelike avatars in real time, TaoAvatar is setting new standards for digital human technology in AR and beyond.
Key features include:
- Photorealistic, topology-consistent 3D full-body avatars
- Real-time rendering at up to 90 FPS on mobile and AR devices
- Advanced 3D Gaussian Splatting for detailed, efficient avatar creation
- Lightweight MLP-based model for mobile performance
- Blend shapes for enhanced expression and motion control
- Interactive demos with synchronized speech, expressions, and gestures
- Supported by the comprehensive TalkBody4D multi-view dataset