The Relationship Between Reasoning and Performance in Large Language Models -- o3 (mini) Thinks Harder, Not Longer
Marthe Ballon, Andres Algaba, Vincent Ginis
2025-02-24
Summary
This paper talks about how newer AI language models are getting better at solving math problems without needing to use longer explanations, suggesting they're becoming more efficient in their reasoning process.
What's the problem?
When AI models solve math problems, they often use a method called chain-of-thought, where they explain their steps. As these models improve, it's not clear if they're getting better because they're using longer explanations or because they're thinking more efficiently.
What's the solution?
The researchers compared different versions of AI models, specifically looking at how long their explanations were when solving math problems. They found that newer models like o3-mini could solve problems more accurately than older ones like o1-mini, without needing longer explanations. They also discovered that for all models, longer explanations usually led to less accurate answers, but this effect was smaller in more advanced models.
Why it matters?
This matters because it shows that AI is getting smarter, not just by thinking longer, but by thinking better. It helps us understand how to make AI more efficient and effective, especially for complex tasks like math problems. This could lead to AI systems that can solve problems faster and more accurately, which is important for many real-world applications.
Abstract
Large language models have demonstrated remarkable progress in mathematical reasoning, leveraging chain-of-thought and test-time compute scaling. However, many open questions remain regarding the interplay between reasoning token usage and accuracy gains. In particular, when comparing models across generations, it is unclear whether improved performance results from longer reasoning chains or more efficient reasoning. We systematically analyze chain-of-thought length across o1-mini and o3-mini variants on the Omni-MATH benchmark, finding that o3-mini (m) achieves superior accuracy without requiring longer reasoning chains than o1-mini. Moreover, we show that accuracy generally declines as reasoning chains grow across all models and compute settings, even when controlling for difficulty of the questions. This accuracy drop is significantly smaller in more proficient models, suggesting that new generations of reasoning models use test-time compute more effectively. Finally, we highlight that while o3-mini (h) achieves a marginal accuracy gain over o3-mini (m), it does so by allocating substantially more reasoning tokens across all problems, even the ones that o3-mini (m) can already solve. These findings provide new insights into the relationship between model capability and reasoning length, with implications for efficiency, scaling, and evaluation methodologies.