T3M: Text Guided 3D Human Motion Synthesis from Speech
Wenshuo Peng, Kaipeng Zhang, Sai Qian Zhang
2024-08-26
Summary
This paper presents T3M, a new method for creating 3D human animations based on speech, allowing for more accurate and customizable motion synthesis.
What's the problem?
Current methods for generating 3D animations from speech only use the audio of the speech, which often leads to animations that are not very accurate or flexible. This limitation makes it hard to create lifelike movements that match what is being said.
What's the solution?
The authors introduce T3M, which allows users to guide the animation process using text descriptions in addition to speech. This means that users can have more control over how the characters move and behave, resulting in more diverse and tailored animations. The experiments showed that T3M performs better than existing methods in both measurable results and overall quality.
Why it matters?
This research is important because it enhances how we can create realistic animations for various applications like video games, virtual reality, and movies. By improving the connection between speech and motion, T3M can lead to more engaging and believable character animations, making storytelling in digital media more effective.
Abstract
Speech-driven 3D motion synthesis seeks to create lifelike animations based on human speech, with potential uses in virtual reality, gaming, and the film production. Existing approaches reply solely on speech audio for motion generation, leading to inaccurate and inflexible synthesis results. To mitigate this problem, we introduce a novel text-guided 3D human motion synthesis method, termed T3M. Unlike traditional approaches, T3M allows precise control over motion synthesis via textual input, enhancing the degree of diversity and user customization. The experiment results demonstrate that T3M can greatly outperform the state-of-the-art methods in both quantitative metrics and qualitative evaluations. We have publicly released our code at https://github.com/Gloria2tt/T3M.git{https://github.com/Gloria2tt/T3M.git}