StyleSplat: 3D Object Style Transfer with Gaussian Splatting
Sahil Jain, Avik Kuthiala, Prabhdeep Singh Sethi, Prakanshul Saxena
2024-07-15
Summary
This paper introduces StyleSplat, a new method for applying artistic styles to 3D objects using a technique called Gaussian splatting, allowing for quick and customizable style transfers in 3D scenes.
What's the problem?
Existing methods for transferring styles to 3D objects can be slow and often apply the same style to all objects in a scene. This makes it difficult for artists and developers to create unique, stylized versions of multiple objects at once, limiting creative expression.
What's the solution?
StyleSplat solves this problem by using a lightweight approach that first creates a realistic representation of the scene with 3D Gaussian splatting. It segments individual 3D objects and then applies specific styles from reference images to those objects. By using a nearest-neighbor feature matching technique, it ensures that the applied styles look good and are consistent with the original object shapes. This method allows users to quickly apply different styles to different objects within the same scene.
Why it matters?
This research is important because it enhances the ability to create visually appealing 3D content more efficiently. By allowing for localized style transfer, StyleSplat gives artists and game developers more control over their designs, enabling them to produce diverse and interesting 3D environments that reflect their creative visions.
Abstract
Recent advancements in radiance fields have opened new avenues for creating high-quality 3D assets and scenes. Style transfer can enhance these 3D assets with diverse artistic styles, transforming creative expression. However, existing techniques are often slow or unable to localize style transfer to specific objects. We introduce StyleSplat, a lightweight method for stylizing 3D objects in scenes represented by 3D Gaussians from reference style images. Our approach first learns a photorealistic representation of the scene using 3D Gaussian splatting while jointly segmenting individual 3D objects. We then use a nearest-neighbor feature matching loss to finetune the Gaussians of the selected objects, aligning their spherical harmonic coefficients with the style image to ensure consistency and visual appeal. StyleSplat allows for quick, customizable style transfer and localized stylization of multiple objects within a scene, each with a different style. We demonstrate its effectiveness across various 3D scenes and styles, showcasing enhanced control and customization in 3D creation.