Discovering Influential Neuron Path in Vision Transformers
Yifan Wang, Yifei Liu, Yingdong Shi, Changming Li, Anqi Pang, Sibei Yang, Jingyi Yu, Kan Ren
2025-03-14
Summary
This paper investigates how to understand the inner workings of Vision Transformer models, which are powerful AI systems used for image recognition, by identifying the most important pathways of neurons.
What's the problem?
Vision Transformer models are very complex and act like 'black boxes,' making it difficult to understand how they make decisions. This lack of transparency makes it challenging to trust and improve these models.
What's the solution?
The researchers developed a method to identify the most influential neuron paths within the model. These paths represent the flow of information from the input image to the output decision. By finding these crucial paths, they can better understand how the model processes visual information.
Why it matters?
This work matters because it helps make AI models more transparent and understandable. This understanding can lead to improvements in model design, reliability, and potential applications like model pruning (making the model smaller and more efficient).
Abstract
Vision Transformer models exhibit immense power yet remain opaque to human understanding, posing challenges and risks for practical applications. While prior research has attempted to demystify these models through input attribution and neuron role analysis, there's been a notable gap in considering layer-level information and the holistic path of information flow across layers. In this paper, we investigate the significance of influential neuron paths within vision Transformers, which is a path of neurons from the model input to output that impacts the model inference most significantly. We first propose a joint influence measure to assess the contribution of a set of neurons to the model outcome. And we further provide a layer-progressive neuron locating approach that efficiently selects the most influential neuron at each layer trying to discover the crucial neuron path from input to output within the target model. Our experiments demonstrate the superiority of our method finding the most influential neuron path along which the information flows, over the existing baseline solutions. Additionally, the neuron paths have illustrated that vision Transformers exhibit some specific inner working mechanism for processing the visual information within the same image category. We further analyze the key effects of these neurons on the image classification task, showcasing that the found neuron paths have already preserved the model capability on downstream tasks, which may also shed some lights on real-world applications like model pruning. The project website including implementation code is available at https://foundation-model-research.github.io/NeuronPath/.