3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes
Jan Held, Renaud Vandeghen, Abdullah Hamdi, Adrien Deliege, Anthony Cioppa, Silvio Giancola, Andrea Vedaldi, Bernard Ghanem, Marc Van Droogenbroeck
2024-11-28
Summary
This paper introduces 3D Convex Splatting (3DCS), a new method for creating detailed 3D scenes from images that improves upon older techniques by using smooth convex shapes instead of Gaussian shapes.
What's the problem?
Previous methods for rendering 3D scenes, like 3D Gaussian Splatting, have difficulty accurately capturing sharp edges and flat surfaces. This often leads to a need for many Gaussian shapes, which can take up a lot of memory and make the rendering process slower and less efficient.
What's the solution?
The authors developed 3DCS, which uses smooth convex shapes as building blocks for modeling 3D scenes. This approach allows for better representation of hard edges and complex shapes while using fewer primitives than Gaussian methods. They also created a fast rendering system that works efficiently with these shapes, resulting in quicker and higher-quality image generation.
Why it matters?
This research is significant because it enhances the ability to create realistic 3D scenes from images, which has applications in fields like gaming, virtual reality, and film production. By improving the quality and efficiency of scene reconstruction, 3DCS could set a new standard for how we generate and visualize complex environments.
Abstract
Recent advances in radiance field reconstruction, such as 3D Gaussian Splatting (3DGS), have achieved high-quality novel view synthesis and fast rendering by representing scenes with compositions of Gaussian primitives. However, 3D Gaussians present several limitations for scene reconstruction. Accurately capturing hard edges is challenging without significantly increasing the number of Gaussians, creating a large memory footprint. Moreover, they struggle to represent flat surfaces, as they are diffused in space. Without hand-crafted regularizers, they tend to disperse irregularly around the actual surface. To circumvent these issues, we introduce a novel method, named 3D Convex Splatting (3DCS), which leverages 3D smooth convexes as primitives for modeling geometrically-meaningful radiance fields from multi-view images. Smooth convex shapes offer greater flexibility than Gaussians, allowing for a better representation of 3D scenes with hard edges and dense volumes using fewer primitives. Powered by our efficient CUDA-based rasterizer, 3DCS achieves superior performance over 3DGS on benchmarks such as Mip-NeRF360, Tanks and Temples, and Deep Blending. Specifically, our method attains an improvement of up to 0.81 in PSNR and 0.026 in LPIPS compared to 3DGS while maintaining high rendering speeds and reducing the number of required primitives. Our results highlight the potential of 3D Convex Splatting to become the new standard for high-quality scene reconstruction and novel view synthesis. Project page: convexsplatting.github.io.