SynCity: Training-Free Generation of 3D Worlds
Paul Engstler, Aleksandar Shtedritski, Iro Laina, Christian Rupprecht, Andrea Vedaldi
2025-03-21
Summary
This paper discusses a new way to create 3D virtual worlds using AI, without needing to train the AI on specific examples.
What's the problem?
Creating large and detailed 3D worlds is difficult, and most AI models need a lot of training to do it well.
What's the solution?
The researchers developed a system called SynCity that combines existing AI models to generate 3D worlds tile by tile, giving control over the layout and appearance.
Why it matters?
This work matters because it makes it easier and faster to create realistic and immersive 3D environments for games, movies, and other applications.
Abstract
We address the challenge of generating 3D worlds from textual descriptions. We propose SynCity, a training- and optimization-free approach, which leverages the geometric precision of pre-trained 3D generative models and the artistic versatility of 2D image generators to create large, high-quality 3D spaces. While most 3D generative models are object-centric and cannot generate large-scale worlds, we show how 3D and 2D generators can be combined to generate ever-expanding scenes. Through a tile-based approach, we allow fine-grained control over the layout and the appearance of scenes. The world is generated tile-by-tile, and each new tile is generated within its world-context and then fused with the scene. SynCity generates compelling and immersive scenes that are rich in detail and diversity.