Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist
Zihao Zhou, Shudong Liu, Maizhen Ning, Wei Liu, Jindong Wang, Derek F. Wong, Xiaowei Huang, Qiufeng Wang, Kaizhu Huang
2024-07-13
Summary
This paper introduces MATHCHECK, a new checklist designed to evaluate the mathematical reasoning abilities of large language models (LLMs). It aims to provide a more comprehensive assessment of how well these models can understand and solve various math problems.
What's the problem?
Current methods for evaluating LLMs mostly focus on their ability to solve specific math problems. This can lead to overfitting, where models perform well on tests but struggle in real-world scenarios. There is a need for a better way to assess whether these models truly understand math concepts and can apply them across different tasks.
What's the solution?
MATHCHECK addresses this issue by providing a structured checklist that includes a variety of mathematical reasoning tasks and robustness tests. It helps evaluate how well models can generalize their knowledge to different types of problems. The authors developed two versions of the checklist: MATHCHECK-GSM for textual reasoning and MATHCHECK-GEO for multi-modal reasoning, allowing them to assess over 20 LLMs and 11 multimodal language models (MLLMs). The results showed that while top models like GPT-4o perform well, many others struggled, indicating that MATHCHECK provides a better reflection of true mathematical abilities compared to traditional benchmarks.
Why it matters?
This research is important because it offers a new way to evaluate AI models' mathematical reasoning skills more accurately. By using MATHCHECK, developers can gain deeper insights into how well these models understand math, which can lead to improvements in their design and application in real-world situations, such as education and problem-solving tasks.
Abstract
Exceptional mathematical reasoning ability is one of the key features that demonstrate the power of large language models (LLMs). How to comprehensively define and evaluate the mathematical abilities of LLMs, and even reflect the user experience in real-world scenarios, has emerged as a critical issue. Current benchmarks predominantly concentrate on problem-solving capabilities, which presents a substantial risk of model overfitting and fails to accurately represent genuine mathematical reasoning abilities. In this paper, we argue that if a model really understands a problem, it should be robustly and readily applied across a diverse array of tasks. Motivated by this, we introduce MATHCHECK, a well-designed checklist for testing task generalization and reasoning robustness, as well as an automatic tool to generate checklists efficiently. MATHCHECK includes multiple mathematical reasoning tasks and robustness test types to facilitate a comprehensive evaluation of both mathematical reasoning ability and behavior testing. Utilizing MATHCHECK, we develop MATHCHECK-GSM and MATHCHECK-GEO to assess mathematical textual reasoning and multi-modal reasoning capabilities, respectively, serving as upgraded versions of benchmarks including GSM8k, GeoQA, UniGeo, and Geometry3K. We adopt MATHCHECK-GSM and MATHCHECK-GEO to evaluate over 20 LLMs and 11 MLLMs, assessing their comprehensive mathematical reasoning abilities. Our results demonstrate that while frontier LLMs like GPT-4o continue to excel in various abilities on the checklist, many other model families exhibit a significant decline. Further experiments indicate that, compared to traditional math benchmarks, MATHCHECK better reflects true mathematical abilities and represents mathematical intelligence more linearly, thereby supporting our design. On our MATHCHECK, we can easily conduct detailed behavior analysis to deeply investigate models.