Complementary Reinforcement Learning
Dilxat Muhtar, Jiashun Liu, Wei Gao, Weixun Wang, Shaopan Xiong, Ju Huang, Siran Yang, Wenbo Su, Jiamang Wang, Ling Pan, Bo Zheng
2026-03-19
Summary
This paper introduces a new approach to training AI agents, specifically those built using large language models, to learn more efficiently from their experiences.
What's the problem?
Currently, training these AI agents using reinforcement learning is difficult because they need a lot of trial and error to learn, and they don't effectively remember or reuse what they've learned from previous attempts. Existing methods that try to help agents learn from past experiences often fail because the stored experiences become outdated as the agent gets better and better – the old experiences don't match the agent's current abilities.
What's the solution?
The researchers developed a system called Complementary Reinforcement Learning. It works by having two parts that learn *together*: an 'actor' which is the agent making decisions, and an 'experience extractor' which figures out what experiences are most helpful for the actor to learn from. The actor learns from typical rewards, while the experience extractor learns whether the experiences it provides actually help the actor succeed. This way, the experiences stay relevant as the agent improves, constantly evolving alongside the agent's skills.
Why it matters?
This new method significantly improves how quickly and effectively AI agents can learn, showing a 10% performance boost in simple tasks and working well even when the agent needs to learn multiple tasks. It represents a step forward in building AI agents that can learn efficiently from their experiences, making them more practical and powerful.
Abstract
Reinforcement Learning (RL) has emerged as a powerful paradigm for training LLM-based agents, yet remains limited by low sample efficiency, stemming not only from sparse outcome feedback but also from the agent's inability to leverage prior experience across episodes. While augmenting agents with historical experience offers a promising remedy, existing approaches suffer from a critical weakness: the experience distilled from history is either stored statically or fail to coevolve with the improving actor, causing a progressive misalignment between the experience and the actor's evolving capability that diminishes its utility over the course of training. Inspired by complementary learning systems in neuroscience, we present Complementary RL to achieve seamless co-evolution of an experience extractor and a policy actor within the RL optimization loop. Specifically, the actor is optimized via sparse outcome-based rewards, while the experience extractor is optimized according to whether its distilled experiences demonstrably contribute to the actor's success, thereby evolving its experience management strategy in lockstep with the actor's growing capabilities. Empirically, Complementary RL outperforms outcome-based agentic RL baselines that do not learn from experience, achieving 10% performance improvement in single-task scenarios and exhibits robust scalability in multi-task settings. These results establish Complementary RL as a paradigm for efficient experience-driven agent learning.