OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, Wei Chu
2024-11-08
Summary
This paper presents OpenCoder, a high-quality open-source large language model (LLM) for coding that provides detailed resources for researchers to build and improve similar models.
What's the problem?
While there are many proprietary LLMs for coding, there is a lack of open-access models that are suitable for scientific research. This limits the ability of researchers to replicate results and build upon existing work due to the scarcity of transparent data processing and training protocols.
What's the solution?
The authors developed OpenCoder, which not only matches the performance of leading proprietary models but also serves as an 'open cookbook' for the research community. They released not just the model itself but also the training data, data processing methods, and detailed experimental results. This comprehensive approach allows others to understand how to create high-quality code LLMs and encourages collaboration in the field.
Why it matters?
This research is important because it democratizes access to advanced coding models, enabling more researchers to contribute to the development of AI in programming. By providing all necessary resources openly, OpenCoder fosters innovation and helps improve the quality of AI tools used in coding and software development.
Abstract
Large language models (LLMs) for code have become indispensable in various domains, including code generation, reasoning tasks and agent systems.While open-access code LLMs are increasingly approaching the performance levels of proprietary models, high-quality code LLMs suitable for rigorous scientific investigation, particularly those with reproducible data processing pipelines and transparent training protocols, remain limited. The scarcity is due to various challenges, including resource constraints, ethical considerations, and the competitive advantages of keeping models advanced. To address the gap, we introduce OpenCoder, a top-tier code LLM that not only achieves performance comparable to leading models but also serves as an ``open cookbook'' for the research community. Unlike most prior efforts, we release not only model weights and inference code, but also the reproducible training data, complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols for open scientific research. Through this comprehensive release, we identify the key ingredients for building a top-tier code LLM: (1) code optimized heuristic rules for data cleaning and methods for data deduplication, (2) recall of text corpus related to code and (3) high-quality synthetic data in both annealing and supervised fine-tuning stages. By offering this level of openness, we aim to broaden access to all aspects of a top-tier code LLM, with OpenCoder serving as both a powerful model and an open foundation to accelerate research, and enable reproducible advancements in code AI.