Generative Densification: Learning to Densify Gaussians for High-Fidelity Generalizable 3D Reconstruction
Seungtae Nam, Xiangyu Sun, Gyeongjin Kang, Younggeun Lee, Seungjun Oh, Eunbyung Park
2024-12-12
Summary
This paper talks about Generative Densification, a new method for improving 3D reconstruction by enhancing how Gaussian models represent detailed features in images.
What's the problem?
Current methods for creating 3D models from images often struggle to capture fine details because they use a limited number of Gaussian representations. This can lead to a lack of clarity and accuracy in the final 3D models, especially when it comes to high-frequency details like textures and intricate shapes.
What's the solution?
The authors propose a new approach called Generative Densification, which improves the way Gaussian models are used in 3D reconstruction. Instead of just splitting and cloning existing Gaussian parameters, their method up-samples feature representations from pre-trained models to create more detailed Gaussians all at once. This allows for better representation of fine details while maintaining the overall structure of the model. They tested this method on various tasks and found that it outperformed existing techniques, even with smaller model sizes.
Why it matters?
This research is important because it enhances the ability to create high-quality 3D models from images, which has applications in fields like video games, virtual reality, and computer graphics. By improving how details are captured in 3D reconstructions, Generative Densification can lead to more realistic and visually appealing digital environments.
Abstract
Generalized feed-forward Gaussian models have achieved significant progress in sparse-view 3D reconstruction by leveraging prior knowledge from large multi-view datasets. However, these models often struggle to represent high-frequency details due to the limited number of Gaussians. While the densification strategy used in per-scene 3D Gaussian splatting (3D-GS) optimization can be adapted to the feed-forward models, it may not be ideally suited for generalized scenarios. In this paper, we propose Generative Densification, an efficient and generalizable method to densify Gaussians generated by feed-forward models. Unlike the 3D-GS densification strategy, which iteratively splits and clones raw Gaussian parameters, our method up-samples feature representations from the feed-forward models and generates their corresponding fine Gaussians in a single forward pass, leveraging the embedded prior knowledge for enhanced generalization. Experimental results on both object-level and scene-level reconstruction tasks demonstrate that our method outperforms state-of-the-art approaches with comparable or smaller model sizes, achieving notable improvements in representing fine details.