ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning
Xiaoxuan Wang, Han Zhang, Haixin Wang, Yidan Shi, Ruoyan Li, Kaiqiao Han, Chenyi Tong, Haoran Deng, Renliang Sun, Alexander Taylor, Yanqiao Zhu, Jason Cong, Yizhou Sun, Wei Wang
2026-02-26
Summary
This paper focuses on a new way to train artificial intelligence agents, called agentic reinforcement learning, which lets them tackle complicated tasks by interacting with an environment. However, this method is often unreliable and can easily fail during training.
What's the problem?
Agentic reinforcement learning is promising, but it's currently very unstable. This means training often falls apart, making it hard to use in more complex situations or to figure out the best way to set up the training process. It's like trying to build a tower with shaky blocks – it keeps collapsing before you can finish.
What's the solution?
The researchers created a standardized testing environment called ARLArena to carefully study what causes this instability. They broke down the training process into four key parts and tested each one individually. This helped them understand the core issues and develop a new, more stable training method called SAMPO. SAMPO consistently works well across different tasks, preventing the training from collapsing.
Why it matters?
This work is important because it provides a clearer understanding of how agentic reinforcement learning works and offers a practical solution to make it more reliable. This means we can build more powerful and dependable AI agents that can handle increasingly complex problems, and it gives a solid foundation for future research in this area.
Abstract
Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks. Despite encouraging early results, ARL remains highly unstable, often leading to training collapse. This instability limits scalability to larger environments and longer interaction horizons, and constrains systematic exploration of algorithmic design choices. In this paper, we first propose ARLArena, a stable training recipe and systematic analysis framework that examines training stability in a controlled and reproducible setting. ARLArena first constructs a clean and standardized testbed. Then, we decompose policy gradient into four core design dimensions and assess the performance and stability of each dimension. Through this fine-grained analysis, we distill a unified perspective on ARL and propose SAMPO, a stable agentic policy optimization method designed to mitigate the dominant sources of instability in ARL. Empirically, SAMPO achieves consistently stable training and strong performance across diverse agentic tasks. Overall, this study provides a unifying policy gradient perspective for ARL and offers practical guidance for building stable and reproducible LLM-based agent training pipelines.