AntiLeak-Bench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge
Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, William Yang Wang
2024-12-19
Summary
This paper introduces AntiLeak-Bench, a new system designed to prevent data contamination in evaluating large language models (LLMs). It focuses on creating benchmarks that ensure fair testing by using updated real-world knowledge that hasn't been included in the models' training data.
What's the problem?
Data contamination occurs when test data used to evaluate LLMs is also part of their training data. This can lead to unfair evaluations because the models may already know the answers. Existing methods try to update benchmarks with new data but often still include old information and require a lot of manual effort to maintain.
What's the solution?
The authors propose AntiLeak-Bench, which automatically creates benchmark samples that contain only new information not present in the LLMs' training sets. This ensures that evaluations are contamination-free. They also developed a fully automated process for building and updating these benchmarks without needing human labor, making it easier and cheaper to keep them current.
Why it matters?
This research is important because it improves the reliability of evaluations for language models, ensuring they are tested fairly. By preventing data contamination, AntiLeak-Bench can help researchers and developers better understand how well their models perform and make more accurate comparisons between different systems.
Abstract
Data contamination hinders fair LLM evaluation by introducing test data into newer models' training sets. Existing studies solve this challenge by updating benchmarks with newly collected data. However, they fail to guarantee contamination-free evaluation as the newly collected data may contain pre-existing knowledge, and their benchmark updates rely on intensive human labor. To address these issues, we in this paper propose AntiLeak-Bench, an automated anti-leakage benchmarking framework. Instead of simply using newly collected data, we construct samples with explicitly new knowledge absent from LLMs' training sets, which thus ensures strictly contamination-free evaluation. We further design a fully automated workflow to build and update our benchmark without human labor. This significantly reduces the cost of benchmark maintenance to accommodate emerging LLMs. Through extensive experiments, we highlight that data contamination likely exists before LLMs' cutoff time and demonstrate AntiLeak-Bench effectively overcomes this challenge.