VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
Sijie Cheng, Kechen Fang, Yangyang Yu, Sicheng Zhou, Bohao Li, Ye Tian, Tingguang Li, Lei Han, Yang Liu
2024-10-17
Summary
This paper introduces VidEgoThink, a benchmark designed to evaluate how well AI systems understand videos taken from a first-person perspective, which is important for embodied AI applications.
What's the problem?
As AI technology advances, there is a need for systems that can effectively understand and interact with the world from a first-person viewpoint. However, existing benchmarks do not adequately test this capability, making it difficult to assess how well these systems can perform in real-life scenarios where they need to process egocentric videos.
What's the solution?
To address this issue, the authors developed VidEgoThink, which includes four key tasks: answering questions about videos, planning actions based on video content, identifying objects in the video, and modeling rewards for actions taken. They created this benchmark using an automatic data generation process based on the Ego4D dataset and ensured quality through human review. They tested various AI models using this benchmark and found that even advanced models struggled with understanding egocentric videos.
Why it matters?
This research is significant because it highlights the challenges AI systems face in understanding first-person videos, which are crucial for applications like robotics and virtual reality. By providing a comprehensive evaluation tool like VidEgoThink, researchers can better develop AI systems that mimic human-like understanding and interaction in complex environments.
Abstract
Recent advancements in Multi-modal Large Language Models (MLLMs) have opened new avenues for applications in Embodied AI. Building on previous work, EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating egocentric video understanding capabilities. To bridge the gap between MLLMs and low-level control in Embodied AI, we design four key interrelated tasks: video question-answering, hierarchy planning, visual grounding and reward modeling. To minimize manual annotation costs, we develop an automatic data generation pipeline based on the Ego4D dataset, leveraging the prior knowledge and multimodal capabilities of GPT-4o. Three human annotators then filter the generated data to ensure diversity and quality, resulting in the VidEgoThink benchmark. We conduct extensive experiments with three types of models: API-based MLLMs, open-source image-based MLLMs, and open-source video-based MLLMs. Experimental results indicate that all MLLMs, including GPT-4o, perform poorly across all tasks related to egocentric video understanding. These findings suggest that foundation models still require significant advancements to be effectively applied to first-person scenarios in Embodied AI. In conclusion, VidEgoThink reflects a research trend towards employing MLLMs for egocentric vision, akin to human capabilities, enabling active observation and interaction in the complex real-world environments.