GS^3: Efficient Relighting with Triple Gaussian Splatting
Zoubin Bi, Yixin Zeng, Chong Zeng, Fan Pei, Xiang Feng, Kun Zhou, Hongzhi Wu
2024-10-16
Summary
This paper presents GS^3, a new method for efficiently relighting images using a technique called triple Gaussian splatting, which helps create high-quality images from multiple viewpoints.
What's the problem?
Creating realistic images with different lighting and viewpoints can be challenging, especially when trying to maintain high quality and speed. Existing methods often struggle with complex appearances and can be slow, making it hard to use them in real-time applications.
What's the solution?
The authors introduce a new approach that combines spatial and angular Gaussian representations to better handle light and shadows in images. They use a technique called triple splatting, which allows them to efficiently process information from multiple images taken from different angles. This method includes generating shadows and compensating for other lighting effects to produce realistic images quickly. They tested their method on a variety of image types and found it to be fast and effective, achieving high-quality results.
Why it matters?
This research is significant because it improves how we can create realistic images in real-time, which is important for applications like video games, virtual reality, and film production. By making the process faster and more efficient, GS^3 can help artists and developers create better visual content without sacrificing quality.
Abstract
We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex appearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance. Our code and data are publicly available at https://GSrelight.github.io/.