Scaling Image Tokenizers with Grouped Spherical Quantization
Jiangtao Wang, Zhen Qin, Yifan Zhang, Vincent Tao Hu, Björn Ommer, Rania Briq, Stefan Kesselheim
2024-12-04
Summary
This paper discusses a new method called Grouped Spherical Quantization (GSQ) for improving image tokenizers, which are tools that help AI understand and generate images more efficiently.
What's the problem?
Image tokenizers are important for AI systems that generate images, but many existing methods rely on outdated techniques and do not analyze how well they scale. This can lead to poor performance and inefficiencies when trying to create high-quality images, especially when compressing data to save space.
What's the solution?
The researchers introduced GSQ, which uses a new approach to organize and compress image data. It initializes a spherical codebook and uses lookup regularization to keep the data organized on a spherical surface. This allows for better handling of high-dimensional data while maintaining high quality in image reconstruction. They also examined how different factors, like the size of the codebook and compression ratios, affect the performance of image tokenizers. Their findings show that GSQ can effectively reduce data size while still producing high-quality images.
Why it matters?
This research is significant because it provides a more efficient way for AI systems to handle image data, making it easier to generate high-quality images with less computational power. By improving how image tokenizers work, GSQ can help advance various applications in AI, such as graphic design, video games, and virtual reality, where clear and detailed images are essential.
Abstract
Vision tokenizers have gained a lot of attraction due to their scalability and compactness; previous works depend on old-school GAN-based hyperparameters, biased comparisons, and a lack of comprehensive analysis of the scaling behaviours. To tackle those issues, we introduce Grouped Spherical Quantization (GSQ), featuring spherical codebook initialization and lookup regularization to constrain codebook latent to a spherical surface. Our empirical analysis of image tokenizer training strategies demonstrates that GSQ-GAN achieves superior reconstruction quality over state-of-the-art methods with fewer training iterations, providing a solid foundation for scaling studies. Building on this, we systematically examine the scaling behaviours of GSQ, specifically in latent dimensionality, codebook size, and compression ratios, and their impact on model performance. Our findings reveal distinct behaviours at high and low spatial compression levels, underscoring challenges in representing high-dimensional latent spaces. We show that GSQ can restructure high-dimensional latent into compact, low-dimensional spaces, thus enabling efficient scaling with improved quality. As a result, GSQ-GAN achieves a 16x down-sampling with a reconstruction FID (rFID) of 0.50.