GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality
Taoran Yi, Jiemin Fang, Zanwei Zhou, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Xinggang Wang, Qi Tian
2024-07-01
Summary
This paper talks about GaussianDreamerPro, a new system designed to improve the quality of creating 3D objects from text descriptions. It focuses on making 3D models that are not only detailed but also easy to manipulate for various applications, like animation and games.
What's the problem?
Creating 3D models from text has become popular, but previous methods often produce low-quality results. The generated models can be blurry or lack detail because the process of creating them can lead to uncontrolled growth of the 3D shapes, making them look indeterminate. This is a problem because users need high-quality, realistic models for things like video games and movies.
What's the solution?
To solve this issue, the authors developed GaussianDreamerPro, which binds the creation of 3D shapes (called Gaussians) to a stable geometric framework. This means that as the model generates the object, it evolves in a controlled way, improving both its shape and appearance throughout the process. The final output is a detailed 3D model that can be easily used in various applications like animation and simulation.
Why it matters?
This research is important because it enhances how we create 3D assets from text, making it easier to produce high-quality models quickly. By improving the generation process, GaussianDreamerPro opens up new possibilities for using AI in creative fields such as gaming, film production, and virtual reality, where realistic and manipulable 3D objects are essential.
Abstract
Recently, 3D Gaussian splatting (3D-GS) has achieved great success in reconstructing and rendering real-world scenes. To transfer the high rendering quality to generation tasks, a series of research works attempt to generate 3D-Gaussian assets from text. However, the generated assets have not achieved the same quality as those in reconstruction tasks. We observe that Gaussians tend to grow without control as the generation process may cause indeterminacy. Aiming at highly enhancing the generation quality, we propose a novel framework named GaussianDreamerPro. The main idea is to bind Gaussians to reasonable geometry, which evolves over the whole generation process. Along different stages of our framework, both the geometry and appearance can be enriched progressively. The final output asset is constructed with 3D Gaussians bound to mesh, which shows significantly enhanced details and quality compared with previous methods. Notably, the generated asset can also be seamlessly integrated into downstream manipulation pipelines, e.g. animation, composition, and simulation etc., greatly promoting its potential in wide applications. Demos are available at https://taoranyi.com/gaussiandreamerpro/.