SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes
Cheng-De Fan, Chen-Wei Chang, Yi-Ruei Liu, Jie-Ying Lee, Jiun-Long Huang, Yu-Chee Tseng, Yu-Lun Liu
2024-10-23
Summary
This paper introduces SpectroMotion, a new method for creating 3D images of shiny surfaces that change over time, like reflections on water or metal.
What's the problem?
Previous methods for making 3D images of dynamic scenes with shiny surfaces struggled to capture the details accurately. This is important because shiny surfaces reflect light in complex ways, and if not done correctly, the images can look unrealistic or miss important features.
What's the solution?
The researchers developed SpectroMotion by combining different techniques: they used 3D Gaussian Splatting (3DGS) to create the basic structure of the scene, added physically-based rendering (PBR) to simulate how light interacts with surfaces, and included deformation fields to handle changes over time. They also introduced a method to correct surface details during changes and created a training strategy that improves both the shape and color of the surfaces.
Why it matters?
This work is significant because it allows for more realistic and detailed representations of moving shiny objects in 3D graphics. This can enhance applications in gaming, virtual reality, and film, where lifelike visuals are crucial for immersion and storytelling.
Abstract
We present SpectroMotion, a novel approach that combines 3D Gaussian Splatting (3DGS) with physically-based rendering (PBR) and deformation fields to reconstruct dynamic specular scenes. Previous methods extending 3DGS to model dynamic scenes have struggled to accurately represent specular surfaces. Our method addresses this limitation by introducing a residual correction technique for accurate surface normal computation during deformation, complemented by a deformable environment map that adapts to time-varying lighting conditions. We implement a coarse-to-fine training strategy that significantly enhances both scene geometry and specular color prediction. We demonstrate that our model outperforms prior methods for view synthesis of scenes containing dynamic specular objects and that it is the only existing 3DGS method capable of synthesizing photorealistic real-world dynamic specular scenes, outperforming state-of-the-art methods in rendering complex, dynamic, and specular scenes.