Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs
Wang Wei, Tiankai Yang, Hongjie Chen, Yue Zhao, Franck Dernoncourt, Ryan A. Rossi, Hoda Eldardiry
2025-10-10
Summary
This paper introduces a new way to choose the best large language model (LLM) for a specific task, focusing on making the process more efficient and adaptable in real-world situations.
What's the problem?
Currently, systems often either spend too much money using very powerful LLMs for simple tasks, or they risk getting poor results from cheaper, less capable models. Existing methods for choosing between LLMs usually require knowing the results of *all* models beforehand during a training phase, which isn't realistic. In reality, you only see the result from the model you actually *use*, making it hard to learn which model is best for different situations.
What's the solution?
The researchers developed a system called BaRP, which stands for Bandit-feedback Routing with Preferences. It learns to route tasks to the appropriate LLM by acting like it's constantly testing and learning in real-time, similar to how a gambler tries different slot machines. Importantly, it learns using only the feedback from the model it chooses, just like in a real deployment. It also allows users to adjust how much they value accuracy versus cost, letting them fine-tune the system's behavior without needing to retrain it.
Why it matters?
This work is important because it makes using LLMs more practical and cost-effective. By intelligently routing tasks to the right model, it improves performance and reduces expenses. The system's ability to adapt to changing conditions and user preferences makes it a significant step forward in deploying LLMs at a large scale, and it consistently outperforms existing methods.
Abstract
Efficient use of large language models (LLMs) is critical for deployment at scale: without adaptive routing, systems either overpay for strong models or risk poor performance from weaker ones. Selecting the right LLM for each query is fundamentally an online decision problem: models differ in strengths, prices fluctuate, and users value accuracy and cost differently. Yet most routers are trained offline with labels for all candidate models, an assumption that breaks in deployment, where only the outcome of the chosen model is observed. We bridge this gap with BaRP, a Bandit-feedback Routing with Preferences approach that trains under the same partial-feedback restriction as deployment, while supporting preference-tunable inference: operators can dial the performance/cost trade-off at test time without retraining. Framed as a contextual bandit over prompt features and a user preference vector, our method simulates an online feedback setting during training and adapts its routing decisions to each new prompt, rather than depending on full-information offline supervision. Comprehensive experiments show that our method consistently outperforms strong offline routers by at least 12.46% and the largest LLM by at least 2.45%, and generalizes robustly for unseen tasks.