AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions
Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tuney Zheng, Minghao Liu, Xinyao Niu, Yue Wang, Jian Yang, Jiaheng Liu, Wanjun Zhong, Wangchunshu Zhou, Wenhao Huang, Ge Zhang
2024-10-30
Summary
This paper introduces AutoKaggle, a framework designed to help data scientists automate and improve their work in data science competitions by using a system of specialized AI agents.
What's the problem?
Data science tasks, especially those involving tabular data, can be very complex and time-consuming. Data scientists often face challenges in managing their workflows, which can slow down their progress and make it difficult to compete effectively in data science competitions like those on Kaggle. Traditional methods may not be efficient enough to handle the various stages of data processing, model training, and evaluation.
What's the solution?
The authors propose AutoKaggle, a multi-agent framework that uses different AI agents to handle various tasks in the data science pipeline. Each agent specializes in specific areas, such as cleaning data, selecting models, and tuning parameters. This collaborative system allows for an iterative process where code execution, debugging, and testing occur seamlessly. The framework is customizable, enabling users to intervene at any stage of the process, combining automated intelligence with human expertise. The authors tested AutoKaggle on eight Kaggle competitions and found it to be effective in achieving high validation submission rates and scores.
Why it matters?
This research is important because it shows how AI can enhance the efficiency of data science work. By automating many tasks that data scientists typically perform manually, AutoKaggle can save time and improve the quality of results. This could lead to more successful outcomes in competitions and make data science more accessible to a wider range of people.
Abstract
Data science tasks involving tabular data present complex challenges that require sophisticated problem-solving approaches. We propose AutoKaggle, a powerful and user-centric framework that assists data scientists in completing daily data pipelines through a collaborative multi-agent system. AutoKaggle implements an iterative development process that combines code execution, debugging, and comprehensive unit testing to ensure code correctness and logic consistency. The framework offers highly customizable workflows, allowing users to intervene at each phase, thus integrating automated intelligence with human expertise. Our universal data science toolkit, comprising validated functions for data cleaning, feature engineering, and modeling, forms the foundation of this solution, enhancing productivity by streamlining common tasks. We selected 8 Kaggle competitions to simulate data processing workflows in real-world application scenarios. Evaluation results demonstrate that AutoKaggle achieves a validation submission rate of 0.85 and a comprehensive score of 0.82 in typical data science pipelines, fully proving its effectiveness and practicality in handling complex data science tasks.