OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes
Yukun Huang, Jiwen Yu, Yanning Zhou, Jianan Wang, Xintao Wang, Pengfei Wan, Xihui Liu
2025-10-31
Summary
This paper focuses on creating realistic 3D environments from 2D panoramic images, aiming to make these virtual worlds suitable for advanced graphics techniques like realistic lighting and simulations.
What's the problem?
Currently, creating 3D scenes often involves either painstakingly building them by hand (procedural generation) or trying to 'lift' a 2D image into 3D. Existing 'lifting' methods, particularly those using panoramas, are good at making things *look* nice, but they often don't accurately capture the underlying physical properties of objects – things like how light reflects off surfaces, which is crucial for realistic rendering. They focus on appearance and ignore the actual material properties.
What's the solution?
The researchers developed a system called OmniX. It cleverly reuses the power of existing 2D image generation models, but adapts them to understand and recreate not just the *look* of a panoramic scene, but also the geometry, textures, and physical material properties. They built a special 'adapter' that allows these 2D models to perceive depth and material information from panoramas. To help train their system, they also created a large dataset of high-quality panoramic images with detailed information about the materials and objects within them.
Why it matters?
This work is important because it makes it easier to generate high-quality, physically accurate 3D scenes from 2D images. This opens up possibilities for creating more immersive and realistic virtual worlds for applications like video games, simulations, and virtual reality, where accurate lighting and material properties are essential for a believable experience.
Abstract
There are two prevalent ways to constructing 3D scenes: procedural generation and 2D lifting. Among them, panorama-based 2D lifting has emerged as a promising technique, leveraging powerful 2D generative priors to produce immersive, realistic, and diverse 3D environments. In this work, we advance this technique to generate graphics-ready 3D scenes suitable for physically based rendering (PBR), relighting, and simulation. Our key insight is to repurpose 2D generative models for panoramic perception of geometry, textures, and PBR materials. Unlike existing 2D lifting approaches that emphasize appearance generation and ignore the perception of intrinsic properties, we present OmniX, a versatile and unified framework. Based on a lightweight and efficient cross-modal adapter structure, OmniX reuses 2D generative priors for a broad range of panoramic vision tasks, including panoramic perception, generation, and completion. Furthermore, we construct a large-scale synthetic panorama dataset containing high-quality multimodal panoramas from diverse indoor and outdoor scenes. Extensive experiments demonstrate the effectiveness of our model in panoramic visual perception and graphics-ready 3D scene generation, opening new possibilities for immersive and physically realistic virtual world generation.