ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolution
Long Cui, Weiyun Wang, Jie Shao, Zichen Wen, Gen Luo, Linfeng Zhang, Yanting Zhang, Yu Qiao, Wenhai Wang
2025-10-15
Summary
This paper introduces a new technique called Visual Consistency Learning, or ViCO, to make Multimodal Large Language Models (MLLMs) – which process both text and images – more efficient.
What's the problem?
MLLMs are getting more expensive to use because when they look at images, they break those images down into 'vision tokens'. The more detailed the image, the more tokens are needed, and the more processing power is required. This means analyzing complex images takes a lot of computing resources and time, making them slow and costly.
What's the solution?
The researchers developed a system that intelligently decides how many vision tokens each image *really* needs. Instead of using a fixed number of tokens or just adjusting based on image size, they use multiple 'connectors' that can compress images to different degrees. A 'router' then chooses the best connector for each image based on how complex the image is – a simple image gets fewer tokens, a detailed one gets more, but still potentially fewer than before. They train the model to give similar answers no matter which connector is used, ensuring accuracy isn't sacrificed for speed.
Why it matters?
This work is important because it could significantly reduce the cost of using MLLMs, making them more accessible and practical for a wider range of applications. By reducing the number of vision tokens needed, the models can run faster and require less powerful hardware, paving the way for more efficient AI systems that can understand both text and images.
Abstract
Existing Multimodal Large Language Models (MLLMs) suffer from increased inference costs due to the additional vision tokens introduced by image inputs. In this work, we propose Visual Consistency Learning (ViCO), a novel training algorithm that enables the model to represent images of varying semantic complexities using different numbers of vision tokens. The key idea behind our method is to employ multiple MLP connectors, each with a different image compression ratio, to downsample the vision tokens based on the semantic complexity of the image. During training, we minimize the KL divergence between the responses conditioned on different MLP connectors. At inference time, we introduce an image router, termed Visual Resolution Router (ViR), that automatically selects the appropriate compression rate for each image patch. Compared with existing dynamic high-resolution strategies, which adjust the number of visual tokens based on image resolutions, our method dynamically adapts the number of visual tokens according to semantic complexity. Experimental results demonstrate that our method can reduce the number of vision tokens by up to 50% while maintaining the model's perception, reasoning, and OCR capabilities. We hope this work will contribute to the development of more efficient MLLMs. The code and models will be released to facilitate future research.