EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
Zhili Cheng, Yuge Tu, Ran Li, Shiqi Dai, Jinyi Hu, Shengding Hu, Jiahao Li, Yang Shi, Tianyu Yu, Weize Chen, Lei Shi, Maosong Sun
2025-01-24
Summary
This paper talks about EmbodiedEval, a new way to test how well AI models that understand both language and images (called Multimodal Large Language Models or MLLMs) can perform tasks in virtual 3D environments. It's like creating a virtual obstacle course for AI to see how well they can navigate, interact with objects, and answer questions about their surroundings.
What's the problem?
Current ways of testing these AI models mostly use static images or videos, which doesn't show how well the AI can actually interact with an environment. It's like judging how good someone is at sports by only looking at pictures of them, instead of watching them play. Also, the tests we have for AI that can move around in virtual worlds (embodied AI) are too specific and don't cover enough different types of tasks.
What's the solution?
The researchers created EmbodiedEval, which is like a big, diverse video game for AI. It has 328 different tasks in 125 different 3D scenes. These tasks are split into five types: moving around, interacting with objects, social interaction, answering questions about what things look like, and answering questions about where things are. They tested some of the best AI models using EmbodiedEval and found out that even these advanced AIs are still far behind humans when it comes to doing tasks in a 3D environment.
Why it matters?
This matters because as we try to create AI that can help us in the real world, like robots or virtual assistants, we need to know how well they can actually understand and interact with 3D spaces. EmbodiedEval gives us a way to test this and shows us where our current AI models are falling short. By making their test available to everyone, the researchers are helping other scientists figure out how to make AI better at these real-world tasks. This could lead to smarter robots, more helpful virtual assistants, and AI that can better understand and interact with the world around us.
Abstract
Multimodal Large Language Models (MLLMs) have shown significant advancements, providing a promising future for embodied agents. Existing benchmarks for evaluating MLLMs primarily utilize static images or videos, limiting assessments to non-interactive scenarios. Meanwhile, existing embodied AI benchmarks are task-specific and not diverse enough, which do not adequately evaluate the embodied capabilities of MLLMs. To address this, we propose EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs with embodied tasks. EmbodiedEval features 328 distinct tasks within 125 varied 3D scenes, each of which is rigorously selected and annotated. It covers a broad spectrum of existing embodied AI tasks with significantly enhanced diversity, all within a unified simulation and evaluation framework tailored for MLLMs. The tasks are organized into five categories: navigation, object interaction, social interaction, attribute question answering, and spatial question answering to assess different capabilities of the agents. We evaluated the state-of-the-art MLLMs on EmbodiedEval and found that they have a significant shortfall compared to human level on embodied tasks. Our analysis demonstrates the limitations of existing MLLMs in embodied capabilities, providing insights for their future development. We open-source all evaluation data and simulation framework at https://github.com/thunlp/EmbodiedEval.