Language Models are Symbolic Learners in Arithmetic
Chunyuan Deng, Zhiqi Li, Roy Xie, Ruidi Chang, Hanjie Chen
2024-10-25
Summary
This paper explores how large language models (LLMs) learn arithmetic and suggests that they are primarily symbolic learners, meaning they understand math through symbols rather than through numerical computation.
What's the problem?
Many believe that LLMs struggle with arithmetic because they don't process numbers in the same way humans do. However, there hasn't been much evidence to support this idea. The challenge is to understand how LLMs actually learn and use arithmetic, especially when it comes to recognizing and using partial products (the results of multiplying parts of numbers) during calculations.
What's the solution?
The researchers conducted experiments to see how LLMs handle arithmetic tasks. They found that while LLMs can identify some partial products, they often don't use them effectively. They also discovered that when LLMs break down arithmetic tasks into smaller parts (or subgroups), they tend to learn easier patterns first before tackling more complex ones. This means LLMs learn in a way that starts with simple tasks and gradually moves to harder ones, which is called an easy-to-hard learning approach.
Why it matters?
This research is important because it helps us understand the learning processes of LLMs in arithmetic. By confirming that LLMs are symbolic learners, we can improve how we train these models for better performance in math-related tasks, making them more reliable for applications like tutoring or educational software.
Abstract
Large Language Models (LLMs) are thought to struggle with arithmetic learning due to the inherent differences between language modeling and numerical computation, but concrete evidence has been lacking. This work responds to this claim through a two-side experiment. We first investigate whether LLMs leverage partial products during arithmetic learning. We find that although LLMs can identify some partial products after learning, they fail to leverage them for arithmetic tasks, conversely. We then explore how LLMs approach arithmetic symbolically by breaking tasks into subgroups, hypothesizing that difficulties arise from subgroup complexity and selection. Our results show that when subgroup complexity is fixed, LLMs treat a collection of different arithmetic operations similarly. By analyzing position-level accuracy across different training sizes, we further observe that it follows a U-shaped pattern: LLMs quickly learn the easiest patterns at the first and last positions, while progressively learning the more difficult patterns in the middle positions. This suggests that LLMs select subgroup following an easy-to-hard paradigm during learning. Our work confirms that LLMs are pure symbolic learners in arithmetic tasks and underscores the importance of understanding them deeply through subgroup-level quantification.