DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents
Yansong Ning, Rui Liu, Jun Wang, Kai Chen, Wei Li, Jun Fang, Kan Zheng, Naiqiang Tan, Hao Liu
2025-10-09
Summary
This paper introduces DeepTravel, a new system that uses artificial intelligence to automatically plan trips, including booking transportation and lodging, and finding interesting places to visit.
What's the problem?
Current travel planning AI systems are limited because they rely on pre-written instructions and fixed ways of working, making them inflexible and unable to learn and improve on their own. They also struggle with the limitations of real-world travel booking websites, like inconsistent information or slow responses.
What's the solution?
The researchers created a simulated travel environment with lots of pre-loaded data to allow the AI to practice without being slowed down by real-world issues. They then used a special type of learning called reinforcement learning, where the AI gets rewards for good plans and penalties for bad ones. A key part of their approach is a two-step reward system that first checks if a trip is even possible (like making sure flights and hotels are at reasonable times and locations) and then checks if the details of the trip make sense based on information from booking tools. They also let the AI learn from its mistakes by replaying past failures.
Why it matters?
DeepTravel shows that even relatively small AI models can perform travel planning better than much larger, more powerful models currently available. This means more accessible and effective AI travel planning is possible, potentially leading to better and more personalized travel experiences for everyone.
Abstract
Travel planning (TP) agent has recently worked as an emerging building block to interact with external tools and resources for travel itinerary generation, ensuring enjoyable user experience. Despite its benefits, existing studies rely on hand craft prompt and fixed agent workflow, hindering more flexible and autonomous TP agent. This paper proposes DeepTravel, an end to end agentic reinforcement learning framework for building autonomous travel planning agent, capable of autonomously planning, executing tools, and reflecting on tool responses to explore, verify, and refine intermediate actions in multi step reasoning. To achieve this, we first construct a robust sandbox environment by caching transportation, accommodation and POI data, facilitating TP agent training without being constrained by real world APIs limitations (e.g., inconsistent outputs). Moreover, we develop a hierarchical reward modeling system, where a trajectory level verifier first checks spatiotemporal feasibility and filters unsatisfied travel itinerary, and then the turn level verifier further validate itinerary detail consistency with tool responses, enabling efficient and precise reward service. Finally, we propose the reply augmented reinforcement learning method that enables TP agent to periodically replay from a failures experience buffer, emerging notable agentic capacity. We deploy trained TP agent on DiDi Enterprise Solutions App and conduct comprehensive online and offline evaluations, demonstrating that DeepTravel enables small size LLMs (e.g., Qwen3 32B) to significantly outperform existing frontier LLMs such as OpenAI o1, o3 and DeepSeek R1 in travel planning tasks.