RelBench: A Benchmark for Deep Learning on Relational Databases
Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan E. Lenssen, Yiwen Yuan, Zecheng Zhang, Xinwei He, Jure Leskovec
2024-08-05
Summary
This paper introduces RelBench, a benchmark designed to evaluate deep learning models that work with relational databases. It focuses on how to use graph neural networks to solve predictive tasks more effectively than traditional methods.
What's the problem?
Working with relational databases—where data is organized in tables linked by relationships—can be difficult for machine learning models. Most existing methods require manual feature engineering, which is time-consuming and often leads to errors. This means that building accurate models is challenging and inefficient, especially when dealing with multiple interconnected tables.
What's the solution?
To solve this problem, the authors developed RelBench, which allows researchers to test and improve their models on various tasks using relational databases. They introduced a new approach called Relational Deep Learning (RDL) that uses graph neural networks to automatically learn from the data without needing manual adjustments. This method takes advantage of the relationships between tables to create better predictive models more quickly and accurately.
Why it matters?
This research is important because it simplifies the process of using relational databases for machine learning, making it faster and more effective. By improving how models learn from complex data structures, RelBench can help in many fields, such as finance, healthcare, and social media analysis, where understanding relationships in data is crucial.
Abstract
We present RelBench, a public benchmark for solving predictive tasks over relational databases with graph neural networks. RelBench provides databases and tasks spanning diverse domains and scales, and is intended to be a foundational infrastructure for future research. We use RelBench to conduct the first comprehensive study of Relational Deep Learning (RDL) (Fey et al., 2024), which combines graph neural network predictive models with (deep) tabular models that extract initial entity-level representations from raw tables. End-to-end learned RDL models fully exploit the predictive signal encoded in primary-foreign key links, marking a significant shift away from the dominant paradigm of manual feature engineering combined with tabular models. To thoroughly evaluate RDL against this prior gold-standard, we conduct an in-depth user study where an experienced data scientist manually engineers features for each task. In this study, RDL learns better models whilst reducing human work needed by more than an order of magnitude. This demonstrates the power of deep learning for solving predictive tasks over relational databases, opening up many new research opportunities enabled by RelBench.