Mixture of Nested Experts: Adaptive Processing of Visual Tokens
Gagan Jain, Nidhi Hegde, Aditya Kusupati, Arsha Nagrani, Shyamal Buch, Prateek Jain, Anurag Arnab, Sujoy Paul
2024-07-30
Summary
This paper introduces Mixture of Nested Experts (MoNE), a new model designed to improve how visual data (like images and videos) is processed by using a more efficient approach that takes advantage of the redundancy in visual information.
What's the problem?
Current models for processing visual data, especially Vision Transformers (ViTs), can be very resource-intensive and do not effectively utilize the repetitive information found in images and videos. This leads to higher computational costs and slower processing times, making it harder to scale these models for larger datasets or real-time applications.
What's the solution?
The authors developed MoNE, which uses a nested structure of experts that can handle different levels of complexity in visual data. Instead of processing all visual tokens with the same model, MoNE dynamically selects which 'expert' to use based on the importance of each token. This means that less important or redundant tokens are processed using simpler, faster experts, while more critical tokens are handled by more complex models. As a result, MoNE can achieve similar performance to existing models while significantly reducing the time and resources needed for processing.
Why it matters?
This research is important because it enhances the efficiency of AI models in handling visual data, which can lead to faster and more effective applications in areas like image recognition, video analysis, and real-time processing. By improving how these models work, MoNE could make advanced visual AI technologies more accessible and practical for various industries.
Abstract
The visual medium (images and videos) naturally contains a large amount of information redundancy, thereby providing a great opportunity for leveraging efficiency in processing. While Vision Transformer (ViT) based models scale effectively to large data regimes, they fail to capitalize on this inherent redundancy, leading to higher computational costs. Mixture of Experts (MoE) networks demonstrate scalability while maintaining same inference-time costs, but they come with a larger parameter footprint. We present Mixture of Nested Experts (MoNE), which utilizes a nested structure for experts, wherein individual experts fall on an increasing compute-accuracy curve. Given a compute budget, MoNE learns to dynamically choose tokens in a priority order, and thus redundant tokens are processed through cheaper nested experts. Using this framework, we achieve equivalent performance as the baseline models, while reducing inference time compute by over two-fold. We validate our approach on standard image and video datasets - ImageNet-21K, Kinetics400, and Something-Something-v2. We further highlight MoNE's adaptability by showcasing its ability to maintain strong performance across different inference-time compute budgets on videos, using only a single trained model.