BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, Tim Rocktäschel
2024-11-25
Summary
This paper introduces BALROG, a new benchmark designed to evaluate the reasoning abilities of large language models (LLMs) and vision language models (VLMs) in complex game environments.
What's the problem?
While LLMs and VLMs have impressive knowledge and reasoning skills, they often struggle in dynamic and complicated situations, like those found in games. Current methods for testing these models do not effectively measure their capabilities in handling intricate interactions, long-term planning, and spatial reasoning, which are essential for real-world tasks.
What's the solution?
BALROG addresses this issue by providing a comprehensive set of challenging games that vary in difficulty. The benchmark includes both easy tasks that can be solved quickly by non-expert humans and very difficult tasks that may take years to master. It uses fine-grained metrics to evaluate how well these models perform in different scenarios. The results show that while current models do well on easier games, they face significant challenges with more complex tasks, especially when visual information is involved.
Why it matters?
This research is important because it creates a standardized way to assess the capabilities of LLMs and VLMs in real-world-like situations. By identifying where these models excel and where they struggle, BALROG can help researchers improve AI systems, making them more effective for practical applications in various fields such as gaming, robotics, and decision-making.
Abstract
Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities; however, they still struggle to perform well in complex, dynamic environments. Real-world tasks require handling intricate interactions, advanced spatial reasoning, long-term planning, and continuous exploration of new strategies-areas in which we lack effective methodologies for comprehensively evaluating these capabilities. To address this gap, we introduce BALROG, a novel benchmark designed to assess the agentic capabilities of LLMs and VLMs through a diverse set of challenging games. Our benchmark incorporates a range of existing reinforcement learning environments with varying levels of difficulty, including tasks that are solvable by non-expert humans in seconds to extremely challenging ones that may take years to master (e.g., the NetHack Learning Environment). We devise fine-grained metrics to measure performance and conduct an extensive evaluation of several popular open-source and closed-source LLMs and VLMs. Our findings indicate that while current models achieve partial success in the easier games, they struggle significantly with more challenging tasks. Notably, we observe severe deficiencies in vision-based decision-making, as models perform worse when visual representations of the environments are provided. We release BALROG as an open and user-friendly benchmark to facilitate future research and development in the agentic community.