Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of Experts
Jinqiang Long, Yanqi Dai, Guoxing Yang, Hongpeng Lin, Nanyi Fei, Yizhao Gao, Zhiwu Lu
2024-11-19
Summary
This paper introduces Awaker2.5-VL, a new model designed to improve the performance of Multimodal Large Language Models (MLLMs) by using a mixture of experts to handle various tasks more effectively.
What's the problem?
As MLLMs become more popular for tasks that involve both text and images, they face challenges due to the differences in how data from various tasks is represented. Simply combining all this data can lead to conflicts, known as 'multi-task conflict,' which can reduce the model's overall performance across different tasks.
What's the solution?
To solve this issue, the authors propose Awaker2.5-VL, which uses a Mixture of Experts (MoE) architecture. This means that instead of using one model for everything, Awaker2.5-VL activates different 'experts' within the model depending on the task at hand, allowing it to specialize in specific areas. Additionally, each expert is designed using a low-rank adaptation (LoRA) structure to make training and inference faster and more efficient. The paper includes extensive experiments showing that Awaker2.5-VL performs well on various benchmarks.
Why it matters?
This research is significant because it addresses a major challenge in developing effective AI models that can handle multiple types of tasks simultaneously. By improving how MLLMs work with diverse data, Awaker2.5-VL could enhance applications in areas like visual question answering and object detection, making AI tools more useful in real-world scenarios.
Abstract
As the research of Multimodal Large Language Models (MLLMs) becomes popular, an advancing MLLM model is typically required to handle various textual and visual tasks (e.g., VQA, Detection, OCR, and ChartQA) simultaneously for real-world applications. However, due to the significant differences in representation and distribution among data from various tasks, simply mixing data of all tasks together leads to the well-known``multi-task conflict" issue, resulting in performance degradation across various tasks. To address this issue, we propose Awaker2.5-VL, a Mixture of Experts~(MoE) architecture suitable for MLLM, which acquires the multi-task capabilities through multiple sparsely activated experts. To speed up the training and inference of Awaker2.5-VL, each expert in our model is devised as a low-rank adaptation (LoRA) structure. Extensive experiments on multiple latest benchmarks demonstrate the effectiveness of Awaker2.5-VL. The code and model weight are released in our Project Page: https://github.com/MetabrainAGI/Awaker.