LIMO: Less is More for Reasoning
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, Pengfei Liu
2025-02-06
Summary
This paper talks about LIMO, a new approach that shows large language models can learn to solve complex reasoning problems, like math, with very few examples. It challenges the idea that models need huge amounts of training data to perform well on difficult tasks.
What's the problem?
Most people believe that teaching AI to solve hard reasoning problems requires a lot of training data, often tens or hundreds of thousands of examples. This makes training expensive and time-consuming, and it doesn't always lead to models that can generalize well to new problems.
What's the solution?
The researchers developed LIMO, which uses only 817 carefully chosen training examples to teach the model how to solve complex math problems. By focusing on high-quality examples that act as 'cognitive templates,' LIMO shows the model how to apply its pre-existing knowledge from earlier training. This approach resulted in much better performance compared to older methods that used far more data.
Why it matters?
This research is important because it shows that AI models can achieve strong reasoning abilities with much less training data if the examples are chosen wisely. This makes training faster, cheaper, and more efficient while also improving the model's ability to handle new types of problems. It could change how we think about teaching AI to reason.
Abstract
We present a fundamental discovery that challenges our understanding of how complex reasoning emerges in large language models. While conventional wisdom suggests that sophisticated reasoning tasks demand extensive training data (>100,000 examples), we demonstrate that complex mathematical reasoning abilities can be effectively elicited with surprisingly few examples. Through comprehensive experiments, our proposed model LIMO demonstrates unprecedented performance in mathematical reasoning. With merely 817 curated training samples, LIMO achieves 57.1% accuracy on AIME and 94.8% on MATH, improving from previous SFT-based models' 6.5% and 59.2% respectively, while only using 1% of the training data required by previous approaches. LIMO demonstrates exceptional out-of-distribution generalization, achieving 40.5% absolute improvement across 10 diverse benchmarks, outperforming models trained on 100x more data, challenging the notion that SFT leads to memorization rather than generalization. Based on these results, we propose the Less-Is-More Reasoning Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning capabilities can emerge through minimal but precisely orchestrated demonstrations of cognitive processes. This hypothesis posits that the elicitation threshold for complex reasoning is determined by two key factors: (1) the completeness of the model's encoded knowledge foundation during pre-training, and (2) the effectiveness of post-training examples as "cognitive templates" that show the model how to utilize its knowledge base to solve complex reasoning tasks. To facilitate reproducibility and future research in data-efficient reasoning, we release LIMO as a comprehensive open-source suite at https://github.com/GAIR-NLP/LIMO.