Accelerating Vision Transformers with Adaptive Patch Sizes
Rohan Choudhury, JungEun Kim, Jinhyung Park, Eunho Yang, László A. Jeni, Kris M. Kitani
2025-10-23
Summary
This paper introduces a new way to process images with Vision Transformers, called Adaptive Patch Transformers (APT), that makes them faster and more efficient.
What's the problem?
Traditional Vision Transformers treat all parts of an image the same, breaking it down into equally sized squares. This works okay, but when dealing with high-resolution images, it creates a very long list of things for the computer to process, slowing everything down. Some parts of an image are simple and don't need as much detail, while others are complex and do.
What's the solution?
APT solves this by intelligently deciding how to break down the image. It uses larger squares for simpler areas and smaller squares for more detailed areas. This reduces the overall number of pieces the computer needs to look at, making processing much faster. Importantly, it can be added to existing Vision Transformers without needing to retrain them from scratch, and it quickly adjusts to the new method with minimal training.
Why it matters?
This is important because it makes powerful image processing technology much more practical. By speeding up both training and using these models, APT allows for faster development and deployment of applications like visual question answering, object detection, and image segmentation, especially when working with high-resolution images where speed is crucial.
Abstract
Vision Transformers (ViTs) partition input images into uniformly sized patches regardless of their content, resulting in long input sequence lengths for high-resolution images. We present Adaptive Patch Transformers (APT), which addresses this by using multiple different patch sizes within the same image. APT reduces the total number of input tokens by allocating larger patch sizes in more homogeneous areas and smaller patches in more complex ones. APT achieves a drastic speedup in ViT inference and training, increasing throughput by 40% on ViT-L and 50% on ViT-H while maintaining downstream performance, and can be applied to a previously fine-tuned ViT, converging in as little as 1 epoch. It also significantly reduces training and inference time without loss of performance in high-resolution dense visual tasks, achieving up to 30\% faster training and inference in visual QA, object detection, and semantic segmentation.