FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement
Ian Huang, Yanan Bao, Karen Truong, Howard Zhou, Cordelia Schmid, Leonidas Guibas, Alireza Fathi
2025-03-26
Summary
This paper is about making AI better at placing objects in 3D scenes by combining its understanding of the real world with its ability to reason about geometry.
What's the problem?
AI models that can understand language and images are good at understanding what objects are and how they relate to each other, but they often struggle with the geometric details of placing those objects in a 3D scene.
What's the solution?
The researchers developed a new system called FirePlace that uses AI to extract geometric information from the scene and then uses that information to guide the placement of objects in a way that makes sense both geometrically and in terms of common sense.
Why it matters?
This work matters because it can lead to AI systems that can create more realistic and believable 3D scenes, which is useful for things like video games, virtual reality, and architectural design.
Abstract
Scene generation with 3D assets presents a complex challenge, requiring both high-level semantic understanding and low-level geometric reasoning. While Multimodal Large Language Models (MLLMs) excel at semantic tasks, their application to 3D scene generation is hindered by their limited grounding on 3D geometry. In this paper, we investigate how to best work with MLLMs in an object placement task. Towards this goal, we introduce a novel framework, FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the extraction of relevant geometric details from the 3D scene, (2) constructing and solving geometric constraints on the extracted low-level geometry, and (3) pruning for final placements that conform to common sense. By combining geometric reasoning with real-world understanding of MLLMs, our method can propose object placements that satisfy both geometric constraints as well as high-level semantic common-sense considerations. Our experiments show that these capabilities allow our method to place objects more effectively in complex scenes with intricate geometry, surpassing the quality of prior work.