Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System
Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, Maosong Sun
2024-10-13
Summary
This paper presents Optima, a new framework designed to improve the effectiveness and efficiency of multi-agent systems that use large language models (LLMs) for collaborative problem-solving.
What's the problem?
Multi-agent systems based on LLMs face several challenges, including inefficient communication between agents, difficulties in scaling up the system, and ineffective methods for updating the parameters that control how agents work together. These issues can hinder their ability to work as a team and solve complex problems effectively.
What's the solution?
To tackle these challenges, the authors developed Optima, which enhances communication efficiency and task effectiveness in LLM-based multi-agent systems. The framework uses an iterative process where agents generate responses, rank them, select the best options, and then train on these selections. It incorporates various reinforcement learning (RL) techniques to find the best balance between performance and efficiency. By using methods inspired by Monte Carlo Tree Search, Optima allows agents to explore different ways of interacting, leading to better collaboration. The results showed significant improvements in performance on tasks that require heavy information exchange, achieving up to 2.8 times better performance while using less than 10% of the tokens typically required.
Why it matters?
This research is important because it demonstrates how to make multi-agent systems more efficient and effective, which can lead to better AI applications in areas like automated customer service, collaborative robotics, and complex problem-solving scenarios. By addressing fundamental issues in how these systems operate, Optima paves the way for scalable and powerful AI solutions.
Abstract
Large Language Model (LLM) based multi-agent systems (MAS) show remarkable potential in collaborative problem-solving, yet they still face critical challenges: low communication efficiency, poor scalability, and a lack of effective parameter-updating optimization methods. We present Optima, a novel framework that addresses these issues by significantly enhancing both communication efficiency and task effectiveness in LLM-based MAS through LLM training. Optima employs an iterative generate, rank, select, and train paradigm with a reward function balancing task performance, token efficiency, and communication readability. We explore various RL algorithms, including Supervised Fine-Tuning, Direct Preference Optimization, and their hybrid approaches, providing insights into their effectiveness-efficiency trade-offs. We integrate Monte Carlo Tree Search-inspired techniques for DPO data generation, treating conversation turns as tree nodes to explore diverse interaction paths. Evaluated on common multi-agent tasks, including information-asymmetric question answering and complex reasoning, Optima shows consistent and substantial improvements over single-agent baselines and vanilla MAS based on Llama 3 8B, achieving up to 2.8x performance gain with less than 10\% tokens on tasks requiring heavy information exchange. Moreover, Optima's efficiency gains open new possibilities for leveraging inference-compute more effectively, leading to improved inference-time scaling laws. By addressing fundamental challenges in LLM-based MAS, Optima shows the potential towards scalable, efficient, and effective MAS (https://chenweize1998.github.io/optima-project-page).