SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding
Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiaogang Wang, Hongsheng Li, Lewei Lu, Jifeng Dai
2024-12-16
Summary
This paper talks about SynerGen-VL, a new type of model that helps computers understand and generate images more effectively by using a simpler design without needing complex encoders.
What's the problem?
Many existing models for understanding and generating images are complicated and difficult to train. They often require a lot of resources and have complex architectures, making it hard to scale them up for larger tasks. This complexity can hinder the development of effective multimodal models that combine text and images.
What's the solution?
SynerGen-VL proposes a simpler approach by using an encoder-free design that integrates both image understanding and generation. It introduces two key innovations: the token folding mechanism, which compresses image data to make it easier to process, and a vision-expert-based training strategy that helps the model learn better from images. This allows SynerGen-VL to achieve high performance while keeping the model size smaller than other similar models.
Why it matters?
This research is important because it paves the way for more efficient AI systems that can handle both text and images seamlessly. By simplifying the training process and reducing resource requirements, SynerGen-VL can lead to advancements in various applications such as content creation, virtual reality, and interactive media, making these technologies more accessible.
Abstract
The remarkable success of Large Language Models (LLMs) has extended to the multimodal domain, achieving outstanding performance in image understanding and generation. Recent efforts to develop unified Multimodal Large Language Models (MLLMs) that integrate these capabilities have shown promising results. However, existing approaches often involve complex designs in model architecture or training pipeline, increasing the difficulty of model training and scaling. In this paper, we propose SynerGen-VL, a simple yet powerful encoder-free MLLM capable of both image understanding and generation. To address challenges identified in existing encoder-free unified MLLMs, we introduce the token folding mechanism and the vision-expert-based progressive alignment pretraining strategy, which effectively support high-resolution image understanding while reducing training complexity. After being trained on large-scale mixed image-text data with a unified next-token prediction objective, SynerGen-VL achieves or surpasses the performance of existing encoder-free unified MLLMs with comparable or smaller parameter sizes, and narrows the gap with task-specific state-of-the-art models, highlighting a promising path toward future unified MLLMs. Our code and models shall be released.