UniTok: A Unified Tokenizer for Visual Generation and Understanding
Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, Xiaojuan Qi
2025-02-28
Summary
This paper talks about UniTok, a new way to help computers understand and create images using the same system. It's like teaching a computer to both read and write pictures using a single language.
What's the problem?
Usually, computers use different systems to understand images and to create them. This makes it hard to combine these abilities efficiently. It's like having two separate dictionaries - one for reading and one for writing - which can lead to confusion and inefficiency.
What's the solution?
The researchers created UniTok, which uses a clever method called multi-codebook quantization. This is like giving the computer a super-dictionary that can handle both reading and writing images. UniTok breaks down the image information into smaller, manageable pieces, allowing the computer to work with images more effectively for both understanding and creating them.
Why it matters?
This matters because it makes computers better at handling images in general. UniTok performs better than previous methods in both understanding and generating images. For example, it's better at recreating images accurately and can understand images without being specifically trained for certain tasks. This could lead to more efficient and capable AI systems for things like image recognition, creating artwork, or even helping with medical imaging.
Abstract
The representation disparity between visual generation and understanding imposes a critical gap in integrating these capabilities into a single framework. To bridge this gap, we introduce UniTok, a discrete visual tokenizer that encodes fine-grained details for generation while also capturing high-level semantics for understanding. Despite recent studies have shown that these objectives could induce loss conflicts in training, we reveal that the underlying bottleneck stems from limited representational capacity of discrete tokens. We address this by introducing multi-codebook quantization, which divides vector quantization with several independent sub-codebooks to expand the latent feature space, while avoiding training instability caused by overlarge codebooks. Our method significantly raises the upper limit of unified discrete tokenizers to match or even surpass domain-specific continuous tokenizers. For instance, UniTok achieves a remarkable rFID of 0.38 (versus 0.87 for SD-VAE) and a zero-shot accuracy of 78.6% (versus 76.2% for CLIP) on ImageNet. Our code is available at https://github.com/FoundationVision/UniTok.