Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu
2024-07-16
Summary
This paper introduces Spider2-V, a new benchmark designed to evaluate how well multimodal agents can automate complex data science and engineering workflows using real-world tasks.
What's the problem?
Data science and engineering involve many steps and tools, such as data storage and processing, which are often complicated and require human intervention. Current AI models struggle to automate these tasks effectively, especially when they need to interact with graphical user interfaces (GUIs) in professional software applications.
What's the solution?
Spider2-V provides a structured way to test multimodal agents on 494 real-world tasks that require both coding and GUI management. The benchmark includes 20 enterprise-level applications and focuses on how well these agents can perform tasks by writing code and controlling software interfaces. The researchers created automatic configurations for setting up these tasks and developed metrics to evaluate the agents' performance. Despite the advanced capabilities of current models, the study found that they only succeeded in fully automating workflows about 14% of the time.
Why it matters?
This research is significant because it highlights the challenges faced by AI in automating complex workflows in data science and engineering. By establishing Spider2-V as a benchmark, it sets the stage for future improvements in AI technology, making it possible for agents to eventually handle more sophisticated tasks independently. This could lead to greater efficiency in data analysis and make these powerful tools more accessible to a wider range of users.
Abstract
Data science and engineering workflows often span multiple stages, from warehousing to orchestration, using tools like BigQuery, dbt, and Airbyte. As vision language models (VLMs) advance in multimodal understanding and code generation, VLM-based agents could potentially automate these workflows by generating SQL queries, Python code, and GUI operations. This automation can improve the productivity of experts while democratizing access to large-scale data analysis. In this paper, we introduce Spider2-V, the first multimodal agent benchmark focusing on professional data science and engineering workflows, featuring 494 real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. These tasks, derived from real-world use cases, evaluate the ability of a multimodal agent to perform data-related tasks by writing code and managing the GUI in enterprise data software systems. To balance realistic simulation with evaluation simplicity, we devote significant effort to developing automatic configurations for task setup and carefully crafting evaluation metrics for each task. Furthermore, we supplement multimodal agents with comprehensive documents of these enterprise data software systems. Our empirical evaluation reveals that existing state-of-the-art LLM/VLM-based agents do not reliably automate full data workflows (14.0% success). Even with step-by-step guidance, these agents still underperform in tasks that require fine-grained, knowledge-intensive GUI actions (16.2%) and involve remote cloud-hosted workspaces (10.6%). We hope that Spider2-V paves the way for autonomous multimodal agents to transform the automation of data science and engineering workflow. Our code and data are available at https://spider2-v.github.io.