RayGauss: Volumetric Gaussian-Based Ray Casting for Photorealistic Novel View Synthesis
Hugo Blanc, Jean-Emmanuel Deschaud, Alexis Paljic
2024-08-08
Summary
This paper introduces RayGauss, a new method for creating photorealistic images from different viewpoints using a technique called volumetric Gaussian-based ray casting.
What's the problem?
While recent methods for rendering 3D images have improved significantly, they often struggle with certain issues. Traditional approaches like Neural Radiance Fields (NeRF) can be slow and may produce visible artifacts in the images. Additionally, existing methods that use Gaussian splatting can lead to quality problems when rendering complex scenes, especially when the Gaussian points are not evenly spaced.
What's the solution?
RayGauss addresses these challenges by using a new way to combine Gaussian functions with ray casting. It develops a method that allows for more accurate and efficient rendering by integrating color and density information from the scene in a way that adapts to its structure. This is achieved through a special algorithm that processes the scene in layers (or slabs), which helps avoid common visual artifacts. The result is high-quality images rendered at a speed of 25 frames per second on standard datasets.
Why it matters?
This research is important because it enhances the ability to create realistic 3D images quickly, which has applications in areas like virtual reality, video games, and film production. By improving how we render complex scenes, RayGauss can lead to better visual experiences and more efficient workflows in digital content creation.
Abstract
Differentiable volumetric rendering-based methods made significant progress in novel view synthesis. On one hand, innovative methods have replaced the Neural Radiance Fields (NeRF) network with locally parameterized structures, enabling high-quality renderings in a reasonable time. On the other hand, approaches have used differentiable splatting instead of NeRF's ray casting to optimize radiance fields rapidly using Gaussian kernels, allowing for fine adaptation to the scene. However, differentiable ray casting of irregularly spaced kernels has been scarcely explored, while splatting, despite enabling fast rendering times, is susceptible to clearly visible artifacts. Our work closes this gap by providing a physically consistent formulation of the emitted radiance c and density {\sigma}, decomposed with Gaussian functions associated with Spherical Gaussians/Harmonics for all-frequency colorimetric representation. We also introduce a method enabling differentiable ray casting of irregularly distributed Gaussians using an algorithm that integrates radiance fields slab by slab and leverages a BVH structure. This allows our approach to finely adapt to the scene while avoiding splatting artifacts. As a result, we achieve superior rendering quality compared to the state-of-the-art while maintaining reasonable training times and achieving inference speeds of 25 FPS on the Blender dataset. Project page with videos and code: https://raygauss.github.io/