MatSpray: Fusing 2D Material World Knowledge on 3D Geometry
Philipp Langsteiner, Jan-Niklas Dihlmann, Hendrik P. A. Lensch
2025-12-23
Summary
This paper presents a new method for adding realistic material properties, like color, roughness, and metallic shine, to 3D models created from images. It aims to make these 3D scenes look more real and easier to work with for things like video games and movies.
What's the problem?
Creating 3D models for games and films requires a lot of manual work, especially when it comes to defining how materials look. While we can now reconstruct the *shape* of a scene from photos pretty well, getting the material details right – how light bounces off surfaces – is still difficult. Existing methods struggle to accurately recreate materials when you change the lighting in a scene. Also, while AI can predict material properties from 2D images, getting those 2D maps to fit correctly onto a 3D shape is a challenge.
What's the solution?
The researchers combined a few different techniques. First, they used a method called Gaussian Splatting to build the 3D shape from images. Then, they used an AI model to *predict* the material properties (albedo, roughness, metallicity) for each part of the scene, creating 2D maps. They then transferred these material properties onto the 3D model, either by adjusting the model to match the predicted look or by directly applying the properties to the 3D shape using a technique called Gaussian ray tracing. Finally, they added a small 'neural network' step to refine the details and make sure everything looks consistent from all angles.
Why it matters?
This work is important because it makes creating realistic 3D scenes much faster and easier. By automating the process of adding material properties, artists and developers can spend less time on tedious tasks and more time on creative work. The resulting 3D models are also more flexible, as they can be easily relit without losing their realistic appearance, which is crucial for high-quality visuals in games, movies, and other applications.
Abstract
Manual modeling of material parameters and 3D geometry is a time consuming yet essential task in the gaming and film industries. While recent advances in 3D reconstruction have enabled accurate approximations of scene geometry and appearance, these methods often fall short in relighting scenarios due to the lack of precise, spatially varying material parameters. At the same time, diffusion models operating on 2D images have shown strong performance in predicting physically based rendering (PBR) properties such as albedo, roughness, and metallicity. However, transferring these 2D material maps onto reconstructed 3D geometry remains a significant challenge. We propose a framework for fusing 2D material data into 3D geometry using a combination of novel learning-based and projection-based approaches. We begin by reconstructing scene geometry via Gaussian Splatting. From the input images, a diffusion model generates 2D maps for albedo, roughness, and metallic parameters. Any existing diffusion model that can convert images or videos to PBR materials can be applied. The predictions are further integrated into the 3D representation either by optimizing an image-based loss or by directly projecting the material parameters onto the Gaussians using Gaussian ray tracing. To enhance fine-scale accuracy and multi-view consistency, we further introduce a light-weight neural refinement step (Neural Merger), which takes ray-traced material features as input and produces detailed adjustments. Our results demonstrate that the proposed methods outperform existing techniques in both quantitative metrics and perceived visual realism. This enables more accurate, relightable, and photorealistic renderings from reconstructed scenes, significantly improving the realism and efficiency of asset creation workflows in content production pipelines.