3D Scene Understanding Through Local Random Access Sequence Modeling
Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh, Jared Watrous, Honglin Chen, Khai Loong Aw, Daniel L. K. Yamins
2025-04-08
Summary
This paper talks about a new AI method called LRAS that helps computers understand and edit 3D scenes from single photos, like figuring out how a room would look from different angles or moving objects around in a picture while keeping everything consistent.
What's the problem?
Current AI tools struggle to keep objects and scenes looking right when editing 3D views or moving things around, especially in complex real-world photos where details matter.
What's the solution?
LRAS breaks down images into small puzzle-like pieces, learns how they connect in 3D space using motion clues (like how objects move between views), and edits scenes by randomly working on different pieces while keeping the whole picture consistent.
Why it matters?
This helps create better AR/VR experiences, robot vision systems, and photo editing tools that can realistically modify 3D scenes without needing special cameras or multiple photos.
Abstract
3D scene understanding from single images is a pivotal problem in computer vision with numerous downstream applications in graphics, augmented reality, and robotics. While diffusion-based modeling approaches have shown promise, they often struggle to maintain object and scene consistency, especially in complex real-world scenarios. To address these limitations, we propose an autoregressive generative approach called Local Random Access Sequence (LRAS) modeling, which uses local patch quantization and randomly ordered sequence generation. By utilizing optical flow as an intermediate representation for 3D scene editing, our experiments demonstrate that LRAS achieves state-of-the-art novel view synthesis and 3D object manipulation capabilities. Furthermore, we show that our framework naturally extends to self-supervised depth estimation through a simple modification of the sequence design. By achieving strong performance on multiple 3D scene understanding tasks, LRAS provides a unified and effective framework for building the next generation of 3D vision models.