Baichuan-Omni Technical Report
Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, Song Chen, Xu Li, Da Pan, Shusen Zhang, Xin Wu, Zheng Liang, Jun Liu, Tao Zhang, Keer Lu, Yaqi Zhao, Yanjun Shen, Fan Yang
2024-10-14
Summary
This paper introduces Baichuan-Omni, an open-source multimodal large language model (MLLM) that can process and analyze text, images, audio, and video simultaneously, providing a rich interactive experience.
What's the problem?
While existing models like GPT-4o perform well in multimodal tasks, there is a lack of high-performing open-source alternatives that can handle multiple types of data (like text and images) at the same time. This gap limits accessibility for researchers and developers who want to use such technologies without relying on proprietary systems.
What's the solution?
Baichuan-Omni is designed to address this issue by using a two-stage training process that enhances its ability to understand and interact with different types of data. The model starts with a base of 7 billion parameters and goes through stages of multimodal alignment and multitask fine-tuning, which helps it effectively manage visual and audio inputs alongside text. This makes it capable of performing well across various tasks involving different data types.
Why it matters?
This research is significant because it provides an open-source tool that can advance the field of multimodal understanding. By making such a powerful model available to the community, it encourages further innovation and development in applications that require complex interactions between text, images, audio, and video.
Abstract
The salient multimodal capabilities and interactive experience of GPT-4o highlight its critical role in practical applications, yet it lacks a high-performing open-source counterpart. In this paper, we introduce Baichuan-Omni, the first open-source 7B Multimodal Large Language Model (MLLM) adept at concurrently processing and analyzing modalities of image, video, audio, and text, while delivering an advanced multimodal interactive experience and strong performance. We propose an effective multimodal training schema starting with 7B model and proceeding through two stages of multimodal alignment and multitask fine-tuning across audio, image, video, and text modal. This approach equips the language model with the ability to handle visual and audio data effectively. Demonstrating strong performance across various omni-modal and multimodal benchmarks, we aim for this contribution to serve as a competitive baseline for the open-source community in advancing multimodal understanding and real-time interaction.