Generative World Explorer
Taiming Lu, Tianmin Shu, Alan Yuille, Daniel Khashabi, Jieneng Chen
2024-11-19
Summary
This paper presents the Generative World Explorer (Genex), a new framework that allows an AI agent to mentally explore and understand large 3D environments without needing to physically navigate them.
What's the problem?
In embodied AI, one major challenge is planning actions based on incomplete information about the environment. Most existing methods require agents to physically explore their surroundings to gather information and update their understanding of the world. This can be inefficient and time-consuming.
What's the solution?
Genex addresses this problem by enabling the agent to perform mental exploration of a 3D world, such as urban scenes, using imagined observations. Instead of relying solely on physical exploration, Genex generates hypothetical scenarios to update its beliefs about the environment. To train this system, the researchers created a synthetic dataset called Genex-DB, which helps the agent learn how to make informed decisions based on both real and imagined information.
Why it matters?
This research is significant because it mimics human cognitive abilities, allowing AI agents to make better decisions without always needing to explore physically. By improving how AI understands and interacts with complex environments, Genex could enhance applications in robotics, virtual reality, and other fields where intelligent navigation and decision-making are crucial.
Abstract
Planning with partial observation is a central challenge in embodied AI. A majority of prior works have tackled this challenge by developing agents that physically explore their environment to update their beliefs about the world state.In contrast, humans can imagine unseen parts of the world through a mental exploration and revise their beliefs with imagined observations. Such updated beliefs can allow them to make more informed decisions, without necessitating the physical exploration of the world at all times. To achieve this human-like ability, we introduce the Generative World Explorer (Genex), an egocentric world exploration framework that allows an agent to mentally explore a large-scale 3D world (e.g., urban scenes) and acquire imagined observations to update its belief. This updated belief will then help the agent to make a more informed decision at the current step. To train Genex, we create a synthetic urban scene dataset, Genex-DB. Our experimental results demonstrate that (1) Genex can generate high-quality and consistent observations during long-horizon exploration of a large virtual physical world and (2) the beliefs updated with the generated observations can inform an existing decision-making model (e.g., an LLM agent) to make better plans.