CodeArena: A Collective Evaluation Platform for LLM Code Generation
Mingzhe Du, Anh Tuan Luu, Bin Ji, Xiaobao Wu, Dong Huang, Terry Yue Zhuo, Qian Liu, See-Kiong Ng
2025-03-04
Summary
This paper talks about CodeArena, an online platform designed to evaluate how well AI models can generate computer code. It uses a unique system to compare different models fairly and improve the way their performance is measured.
What's the problem?
AI models are becoming very good at generating computer code, but there are problems with how their abilities are tested. Current evaluation methods often have issues like bias from repeated test cases, loss of useful data, and limited access to the systems being tested. These problems make it hard to get accurate and fair assessments of the models' coding skills.
What's the solution?
The researchers created CodeArena, a platform that addresses these issues by using a collective evaluation system. This system adjusts the scores of individual models based on how all the models perform together, reducing bias. CodeArena also provides open access to all solutions and test cases, along with tools (APIs) that make it easier to automate the evaluation process. This setup ensures fairer and more efficient testing of AI coding abilities.
Why it matters?
This matters because CodeArena makes it easier to accurately compare and improve AI models that generate code. By solving problems with current evaluation methods, it helps developers create better tools for programming, which can save time and boost productivity in software development.
Abstract
Large Language Models (LLMs) have reshaped code generation by synergizing their exceptional comprehension of natural language and programming syntax, thereby substantially boosting developer productivity. These advancements have prompted numerous efforts to quantitatively evaluate their coding capabilities. However, persistent challenges, such as benchmark leakage, data dissipation, and limited system accessibility, continue to impede a timely and accurate assessment. To address these limitations, we introduce CodeArena, an online evaluation framework tailored for LLM code generation. The key innovation is a collective evaluation mechanism, which dynamically recalibrates individual model scores based on the holistic performance of all participating models, mitigating score biases caused by widespread benchmark leakage. In addition, CodeArena ensures open access to all submitted solutions and test cases and provides automation-friendly APIs to streamline the code evaluation workflow. Our main contributions are: (1) a collective evaluation system for unbiased assessment, (2) a public repository of solutions and test cases, and (3) automation-ready APIs for seamless integration.