GaussianSR: 3D Gaussian Super-Resolution with 2D Diffusion Priors
Xiqian Yu, Hanxin Zhu, Tianyu He, Zhibo Chen
2024-06-17
Summary
This paper presents GaussianSR, a new method for creating high-resolution images from low-resolution inputs using 3D Gaussian Splatting combined with 2D diffusion techniques. It focuses on improving the process of synthesizing new views of objects or scenes.
What's the problem?
Generating high-quality images from low-resolution inputs is difficult because there is often not enough high-resolution data available to train models effectively. Previous methods, like Neural Radiance Fields (NeRF), have tried to solve this but are slow and inefficient, making them less practical for real-time applications.
What's the solution?
To overcome these challenges, the authors developed GaussianSR, which uses a technique called Score Distillation Sampling (SDS) to transfer knowledge from existing 2D image models into a 3D context. This allows the model to create better quality images while maintaining faster rendering speeds. They also introduced two methods to reduce problems caused by randomness in the generative process: first, they limited how much variation could occur during the sampling process, and second, they randomly removed unnecessary data points that didn't contribute to the final image quality. These improvements help ensure that the generated images are clear and detailed.
Why it matters?
This research is significant because it enables the creation of high-quality images from lower-quality inputs, which is important for many applications like virtual reality, gaming, and film production. By making it easier and faster to generate realistic images, GaussianSR can enhance user experiences and broaden the use of 3D graphics in various fields.
Abstract
Achieving high-resolution novel view synthesis (HRNVS) from low-resolution input views is a challenging task due to the lack of high-resolution data. Previous methods optimize high-resolution Neural Radiance Field (NeRF) from low-resolution input views but suffer from slow rendering speed. In this work, we base our method on 3D Gaussian Splatting (3DGS) due to its capability of producing high-quality images at a faster rendering speed. To alleviate the shortage of data for higher-resolution synthesis, we propose to leverage off-the-shelf 2D diffusion priors by distilling the 2D knowledge into 3D with Score Distillation Sampling (SDS). Nevertheless, applying SDS directly to Gaussian-based 3D super-resolution leads to undesirable and redundant 3D Gaussian primitives, due to the randomness brought by generative priors. To mitigate this issue, we introduce two simple yet effective techniques to reduce stochastic disturbances introduced by SDS. Specifically, we 1) shrink the range of diffusion timestep in SDS with an annealing strategy; 2) randomly discard redundant Gaussian primitives during densification. Extensive experiments have demonstrated that our proposed GaussainSR can attain high-quality results for HRNVS with only low-resolution inputs on both synthetic and real-world datasets. Project page: https://chchnii.github.io/GaussianSR/