Jointly Generating Multi-view Consistent PBR Textures using Collaborative Control
Shimon Vainer, Konstantin Kutsy, Dante De Nigris, Ciara Rowles, Slava Elizarov, Simon Donné
2024-10-10
Summary
This paper discusses a new method for generating consistent textures across different views of 3D objects using a technique called Collaborative Control in the context of physically-based rendering (PBR).
What's the problem?
When creating textures for 3D models, it's important that they look the same from different angles. However, many existing methods struggle with this multi-view consistency, meaning that textures can appear misaligned or distorted when viewed from various perspectives. This issue is especially challenging in the Text-to-Texture problem, where the geometric details are already known.
What's the solution?
To tackle this problem, the authors developed a model called Collaborative Control that directly generates PBR textures, including detailed normal maps. This model is unique because it can produce a complete set of PBR textures in a way that maintains consistency across multiple views. The authors also discuss the design choices made to ensure that their model works effectively and demonstrate its effectiveness through various studies and practical applications.
Why it matters?
This research is significant because it improves the way textures are generated for 3D graphics, which is crucial for applications in gaming, virtual reality, and animation. By ensuring that textures remain consistent regardless of the viewing angle, this method can enhance the realism and quality of digital content, making it more visually appealing and immersive.
Abstract
Multi-view consistency remains a challenge for image diffusion models. Even within the Text-to-Texture problem, where perfect geometric correspondences are known a priori, many methods fail to yield aligned predictions across views, necessitating non-trivial fusion methods to incorporate the results onto the original mesh. We explore this issue for a Collaborative Control workflow specifically in PBR Text-to-Texture. Collaborative Control directly models PBR image probability distributions, including normal bump maps; to our knowledge, the only diffusion model to directly output full PBR stacks. We discuss the design decisions involved in making this model multi-view consistent, and demonstrate the effectiveness of our approach in ablation studies, as well as practical applications.