Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization
Zeyuan Liu, Jeonghye Kim, Xufang Luo, Dongsheng Li, Yuqing Yang
2026-02-27
Summary
This paper introduces a new method, called EMPO^2, to help AI agents powered by large language models explore and learn in complex environments, especially when those environments require discovering completely new things.
What's the problem?
Current AI agents using large language models struggle with exploration, meaning they have trouble figuring out what to do when faced with situations they haven't encountered before. They rely heavily on what they've already been taught, and can't easily adapt to find solutions in truly novel environments. Existing methods don't work well when the AI needs to discover new states or situations on its own.
What's the solution?
EMPO^2 tackles this problem by giving the AI agent a 'memory' to store information about its experiences and using a combination of different learning approaches – both learning from its own current actions and from past experiences. This hybrid approach allows the AI to effectively use the memory for exploration, and also ensures it can still function well even without relying on the memory. It's designed to be robust and adaptable.
Why it matters?
This research is important because it significantly improves the ability of AI agents to explore and learn in new and challenging environments. The results show EMPO^2 performs much better than previous methods, and can quickly adapt to new tasks with minimal additional training, paving the way for more capable and generalizable AI systems.
Abstract
Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose Exploratory Memory-Augmented On- and Off-Policy Optimization (EMPO^2), a hybrid RL framework that leverages memory for exploration and combines on- and off-policy updates to make LLMs perform well with memory while also ensuring robustness without it. On ScienceWorld and WebShop, EMPO^2 achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out-of-distribution tests, EMPO^2 demonstrates superior adaptability to new tasks, requiring only a few trials with memory and no parameter updates. These results highlight EMPO^2 as a promising framework for building more exploratory and generalizable LLM-based agents.