SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstruction
Marko Mihajlovic, Sergey Prokudin, Siyu Tang, Robert Maier, Federica Bogo, Tony Tung, Edmond Boyer
2024-09-18
Summary
This paper discusses SplatFields, a new method for creating 3D and 4D models from multiple images that improves the quality of scene reconstruction, especially when there are not many images available.
What's the problem?
Creating accurate 3D models from images can be difficult, especially when there are not enough views (pictures taken from different angles) to work with. Traditional methods often require many images to produce high-quality results, which can be impractical and expensive, especially for dynamic scenes where capturing every angle is challenging.
What's the solution?
The researchers propose a new optimization strategy that improves the existing 3D Gaussian Splatting (3DGS) technique by addressing a key issue: the lack of spatial autocorrelation in splat features. They model these features using an implicit neural field, which helps enhance the quality of the reconstruction even when fewer images are available. Their approach successfully handles both static and dynamic scenes, demonstrating better performance in various testing scenarios.
Why it matters?
This research is significant because it allows for more efficient and effective 3D reconstruction from limited data, making it easier to create realistic models for applications in gaming, virtual reality, and simulations. By improving how we can capture and reconstruct scenes, this work opens up new possibilities for technology in fields like film production, architecture, and scientific visualization.
Abstract
Digitizing 3D static scenes and 4D dynamic events from multi-view images has long been a challenge in computer vision and graphics. Recently, 3D Gaussian Splatting (3DGS) has emerged as a practical and scalable reconstruction method, gaining popularity due to its impressive reconstruction quality, real-time rendering capabilities, and compatibility with widely used visualization tools. However, the method requires a substantial number of input views to achieve high-quality scene reconstruction, introducing a significant practical bottleneck. This challenge is especially severe in capturing dynamic scenes, where deploying an extensive camera array can be prohibitively costly. In this work, we identify the lack of spatial autocorrelation of splat features as one of the factors contributing to the suboptimal performance of the 3DGS technique in sparse reconstruction settings. To address the issue, we propose an optimization strategy that effectively regularizes splat features by modeling them as the outputs of a corresponding implicit neural field. This results in a consistent enhancement of reconstruction quality across various scenarios. Our approach effectively handles static and dynamic cases, as demonstrated by extensive testing across different setups and scene complexities.