MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code
Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li
2024-10-13
Summary
This paper introduces MathCoder2, a method that enhances the mathematical reasoning abilities of large language models (LLMs) by using a specialized dataset of mathematical code and reasoning steps.
What's the problem?
Previous attempts to improve LLMs' math skills often relied on code that was designed for specific fields like engineering or machine learning, which did not focus directly on mathematical reasoning. This limited the models' ability to understand and solve math problems effectively.
What's the solution?
To solve this problem, the authors created a new dataset called MathCode-Pile, which includes 19.2 billion tokens of high-quality mathematical code paired with corresponding reasoning steps. They generated this dataset by collecting math-related web data, textbooks, and synthetic data, then extracting LaTeX expressions (which are used for formatting mathematical equations) along with the necessary conditions and results. Each reasoning step is paired with Python code that reflects the underlying mathematical logic. This approach allows the models to learn not just how to compute answers but also how to understand the reasoning behind them.
Why it matters?
This research is important because it demonstrates that by combining natural language reasoning with precise mathematical code, LLMs can significantly improve their performance in solving math problems. The creation of MathCoder2 provides a valuable resource for future research and development in AI, making it easier for models to tackle complex mathematical tasks and enhancing their overall capabilities.
Abstract
Code has been shown to be effective in enhancing the mathematical reasoning abilities of large language models due to its precision and accuracy. Previous works involving continued mathematical pretraining often include code that utilizes math-related packages, which are primarily designed for fields such as engineering, machine learning, signal processing, or module testing, rather than being directly focused on mathematical reasoning. In this paper, we introduce a novel method for generating mathematical code accompanied with corresponding reasoning steps for continued pretraining. Our approach begins with the construction of a high-quality mathematical continued pretraining dataset by incorporating math-related web data, code using mathematical packages, math textbooks, and synthetic data. Next, we construct reasoning steps by extracting LaTeX expressions, the conditions needed for the expressions, and the results of the expressions from the previously collected dataset. Based on this extracted information, we generate corresponding code to accurately capture the mathematical reasoning process. Appending the generated code to each reasoning step results in data consisting of paired natural language reasoning steps and their corresponding code. Combining this data with the original dataset results in a 19.2B-token high-performing mathematical pretraining corpus, which we name MathCode-Pile. Training several popular base models with this corpus significantly improves their mathematical abilities, leading to the creation of the MathCoder2 family of models. All of our data processing and training code is open-sourced, ensuring full transparency and easy reproducibility of the entire data collection and training pipeline. The code is released at https://github.com/mathllm/MathCoder2 .