SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning
Haiwen Diao, Bo Wan, Xu Jia, Yunzhi Zhuge, Ying Zhang, Huchuan Lu, Long Chen
2024-07-16
Summary
This paper presents SHERL, a new method for transfer learning that aims to efficiently adapt large models to new tasks while using less memory and maintaining high accuracy.
What's the problem?
When adapting large pre-trained models for specific tasks, traditional methods often require a lot of memory and computational resources. This can make it difficult to fine-tune these models, especially in situations where resources are limited. Additionally, existing methods may not fully utilize the knowledge from pre-trained models because they rely on fixed outputs and overlook important relationships between different layers of the model.
What's the solution?
SHERL addresses these challenges by breaking down the adaptation process into two main steps. First, it consolidates intermediate outputs to reduce redundancy and improve compatibility for later use. Then, it uses only a few later layers of the pre-trained model to minimize memory usage while enhancing the model's ability to adapt to new tasks. This approach allows SHERL to achieve performance comparable to or better than traditional methods while using less memory during training.
Why it matters?
This research is significant because it provides a more efficient way to adapt large models for specific tasks, making advanced AI technology more accessible in resource-limited environments. By improving how we can fine-tune these models, SHERL could help accelerate advancements in various fields, such as natural language processing, computer vision, and more.
Abstract
Parameter-efficient transfer learning (PETL) has emerged as a flourishing research field for adapting large pre-trained models to downstream tasks, greatly reducing trainable parameters while grappling with memory challenges during fine-tuning. To address it, memory-efficient series (METL) avoid backpropagating gradients through the large backbone. However, they compromise by exclusively relying on frozen intermediate outputs and limiting the exhaustive exploration of prior knowledge from pre-trained models. Moreover, the dependency and redundancy between cross-layer features are frequently overlooked, thereby submerging more discriminative representations and causing an inherent performance gap (vs. conventional PETL methods). Hence, we propose an innovative METL strategy called SHERL for resource-limited scenarios to decouple the entire adaptation into two successive and complementary processes. In the early route, intermediate outputs are consolidated via an anti-redundancy operation, enhancing their compatibility for subsequent interactions; thereby in the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead and regulate these fairly flexible features into more adaptive and powerful representations for new domains. Extensive ablations on vision-and-language and language-only tasks show that SHERL combines the strengths of both parameter and memory-efficient techniques, performing on-par or better across diverse architectures with lower memory during fine-tuning. Our code is publicly available at: https://github.com/Paranioar/SHERL.