Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists

Summary

This paper introduces a new benchmark for evaluating large language models (LLMs) in the context of data science, specifically focusing on their ability to write feature engineering code.

What's the problem?

Writing feature engineering code is a complex task that requires not only understanding the data but also domain knowledge. Existing methods for evaluating LLMs in this area are limited and often do not effectively measure how well these models can generate useful code for transforming datasets. This gap makes it difficult to assess the true capabilities of LLMs in data science applications.

What's the solution?

The authors propose a benchmark called FeatEng, which allows LLMs to generate Python functions that modify data based on given descriptions. The quality of the generated code is evaluated by measuring how much it improves the performance of an XGBoost model trained on the modified dataset compared to the original. This approach provides a more comprehensive assessment of LLMs' abilities in feature engineering, allowing for better comparisons across different models.

Why it matters?

This research is significant because it helps improve our understanding of how well LLMs can assist in data science tasks. By creating a standardized way to evaluate these models, the study can lead to better tools and techniques for data scientists, ultimately making it easier to analyze and work with complex datasets.