FastCuRL: Curriculum Reinforcement Learning with Progressive Context Extension for Efficient Training R1-like Reasoning Models
Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, Feng Zhang
2025-03-24
Summary
This paper is about making AI models better at reasoning by training them in a smarter way.
What's the problem?
Training AI models to reason well can be slow and require a lot of computing power.
What's the solution?
The researchers developed a new method called FastCuRL that trains the AI in steps, gradually increasing the difficulty of the tasks and the amount of information it has to consider. This makes the training process faster and more efficient.
Why it matters?
This work matters because it can help create AI models that are better at solving complex problems.
Abstract
In this paper, we propose \textsc{FastCuRL}, a simple yet efficient Curriculum Reinforcement Learning approach with context window extending strategy to accelerate the reinforcement learning training efficiency for R1-like reasoning models while enhancing their performance in tackling complex reasoning tasks with long chain-of-thought rationales, particularly with a 1.5B parameter language model. \textsc{FastCuRL} consists of two main procedures: length-aware training data segmentation and context window extension training. Specifically, the former first splits the original training data into three different levels by the input prompt length, and then the latter leverages segmented training datasets with a progressively increasing context window length to train the reasoning model. Experimental results demonstrate that \textsc{FastCuRL}-1.5B-Preview surpasses DeepScaleR-1.5B-Preview across all five datasets (including MATH 500, AIME 2024, AMC 2023, Minerva Math, and OlympiadBench) while only utilizing 50\% of training steps. Furthermore, all training stages for FastCuRL-1.5B-Preview are completed using just a single node with 8 GPUs.