BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, Xun Deng, Zhikai Lei, Miao Zheng, Guoteng Wang, Shuo Zhang, Peng Sun, Rui Zheng, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang
2025-10-23
Summary
This paper focuses on improving how we train large language models, like the ones powering chatbots, using a technique called reinforcement learning. Specifically, it tackles challenges that arise when using older data to train these models, which is a way to make the training process more efficient.
What's the problem?
When you try to train a language model using reinforcement learning with data collected from previous versions of the model (off-policy learning), things can go wrong. The model tends to become too focused on what it already knows, losing its ability to explore new possibilities. This leads to unstable training, where the model’s performance fluctuates wildly, and can even cause the training to completely fail. The core issue is that the training process gets thrown off balance, with negative feedback dominating and a system for preventing the model from exploring new options actually hindering it.
What's the solution?
The researchers developed a new method called BAPO, which stands for Balanced Policy Optimization with Adaptive Clipping. It’s a clever adjustment to existing training techniques. BAPO dynamically changes the limits on how much the model can change its behavior during each training step. This helps to balance the positive and negative feedback, preventing the model from getting stuck and allowing it to continue exploring. Essentially, it makes the training process more stable and efficient.
Why it matters?
This research is important because it allows us to train more powerful and capable language models with less data and more stability. The BAPO method outperforms existing open-source models and even rivals some of the best proprietary systems currently available, like those from Google and OpenAI. This means we can build better AI assistants and tools that are more reliable and effective.
Abstract
Reinforcement learning (RL) has recently become the core paradigm for aligning and strengthening large language models (LLMs). Yet, applying RL in off-policy settings--where stale data from past policies are used for training--improves sample efficiency, but remains challenging: policy entropy declines sharply, optimization often becomes unstable and may even collapse. Through theoretical and empirical analysis, we identify two key insights: (i) an imbalance in optimization, where negative-advantage samples dominate the policy gradient, suppressing useful behaviors and risking gradient explosions; and (ii) the derived Entropy-Clip Rule, which reveals that the fixed clipping mechanism in PPO-like objectives systematically blocks entropy-increasing updates, thereby driving the policy toward over-exploitation at the expense of exploration. Building on these insights, we propose BAlanced Policy Optimization with Adaptive Clipping (BAPO), a simple yet effective method that dynamically adjusts clipping bounds to adaptively re-balance positive and negative contributions, preserve entropy, and stabilize RL optimization. Across diverse off-policy scenarios--including sample replay and partial rollout--BAPO achieves fast, stable, and data-efficient training. On AIME 2024 and AIME 2025 benchmarks, our 7B BAPO model surpasses open-source counterparts such as SkyWork-OR1-7B, while our 32B BAPO model not only achieves state-of-the-art results among models of the same scale but also outperforms leading proprietary systems like o3-mini and Gemini-2.5-Flash-Thinking.