Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Feng Wang, Yaodong Yu, Guoyizhe Wei, Wei Shao, Yuyin Zhou, Alan Yuille, Cihang Xie
2025-02-10
Summary
This paper talks about how breaking images into smaller pieces, called patches, can improve the performance of AI models for understanding images. The researchers found that using smaller patches, even down to individual pixels, helps the models make better predictions.
What's the problem?
When AI models process images, they often break them into larger patches to save computing power. However, this can lead to a loss of important details, making it harder for the models to fully understand the image and perform tasks like classification or segmentation accurately.
What's the solution?
The researchers experimented with different patch sizes and discovered that smaller patches, including pixel-level patches, provide better results. They showed that this approach improves accuracy across various tasks and architectures. By scaling up the number of image tokens to over 50,000 and removing unnecessary components like decoder heads, they achieved high accuracy on benchmarks like ImageNet-1k.
Why it matters?
This matters because it shows a way to build more accurate and efficient AI models for image-related tasks. By focusing on smaller patches, future AI systems could better handle detailed visual information without needing additional complex components, making them both simpler and more powerful.
Abstract
Since the introduction of Vision Transformer (ViT), patchification has long been regarded as a de facto image tokenization approach for plain visual architectures. By compressing the spatial size of images, this approach can effectively shorten the token sequence and reduce the computational cost of ViT-like plain architectures. In this work, we aim to thoroughly examine the information loss caused by this patchification-based compressive encoding paradigm and how it affects visual understanding. We conduct extensive patch size scaling experiments and excitedly observe an intriguing scaling law in patchification: the models can consistently benefit from decreased patch sizes and attain improved predictive performance, until it reaches the minimum patch size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable across different vision tasks, various input scales, and diverse architectures such as ViT and the recent Mamba models. Moreover, as a by-product, we discover that with smaller patches, task-specific decoder heads become less critical for dense prediction. In the experiments, we successfully scale up the visual sequence to an exceptional length of 50,176 tokens, achieving a competitive test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We hope this study can provide insights and theoretical foundations for future works of building non-compressive vision models. Code is available at https://github.com/wangf3014/Patch_Scaling.