FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion
Ziyi Yang, Fanqi Wan, Longguang Zhong, Canbin Huang, Guosheng Liang, Xiaojun Quan
2025-03-07
Summary
This paper talks about FuseChat-3.0, a new set of AI language models that combine the strengths of bigger, more powerful models into smaller, more efficient ones
What's the problem?
Large AI language models are very capable but they're also huge and hard to use in everyday applications. Smaller models are more practical, but they often don't perform as well as the big ones
What's the solution?
The researchers created FuseChat-3.0 by taking the best parts of four large, powerful AI models and teaching them to smaller, more manageable models. They used a two-step process: first, they aligned the smaller models with the larger ones, then they taught the smaller models to make choices similar to the larger models. This process helped the smaller models perform much better on various tasks like following instructions, general knowledge, math, and coding
Why it matters?
This matters because it makes powerful AI language capabilities more accessible and usable in everyday applications. The improved smaller models can now perform almost as well as much larger ones, which could lead to smarter, more capable AI assistants on phones, computers, and other devices without needing super powerful hardware
Abstract
We introduce FuseChat-3.0, a suite of large language models (LLMs) developed by integrating the strengths of heterogeneous source LLMs into more compact target LLMs. Our source models include the powerful Gemma-2-27B-it, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct. For target models, we focus on three widely-used smaller variants-Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen-2.5-7B-Instruct-along with two ultra-compact options, Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct. To leverage the diverse capabilities of these source models, we develop a specialized data construction protocol tailored to various tasks and domains. The FuseChat-3.0 training pipeline consists of two key stages: (1) supervised fine-tuning (SFT) to align the target and source model distributions, and (2) Direct Preference Optimization (DPO) to apply preferences from multiple source LLMs to fine-tune the target model. The resulting FuseChat-3.0 models exhibit significant performance gains across tasks such as instruction following, general knowledge, mathematics, and coding. As illustrated in Figure 1, using Llama-3.1-8B-Instruct as the target model, our fusion approach achieves an average improvement of 6.8 points across 14 benchmarks. Moreover, it demonstrates remarkable gains of 37.1 points and 30.1 points on the instruction-following benchmarks AlpacaEval-2 and Arena-Hard, respectively. Our code, models, and datasets are available at https://github.com/SLIT-AI/FuseChat-3.0.