ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning
Xianming Li, Zongxi Li, Tsz-fung Andrew Lee, Jing Li, Haoran Xie, Qing Li
2026-04-22
Summary
This paper introduces a new method called ShadowPEFT for efficiently adapting large language models (LLMs) to specific tasks, offering an alternative to existing techniques like LoRA.
What's the problem?
Large language models are powerful, but fully retraining them for each new task is incredibly expensive and time-consuming. Existing 'parameter-efficient fine-tuning' methods try to solve this by only updating a small portion of the model's parameters, but they often do this in a way that feels scattered – making small changes to individual parts of the model. This can limit how well the model adapts and doesn't take advantage of the overall structure of the model.
What's the solution?
ShadowPEFT takes a different approach. Instead of tweaking individual weights, it adds a separate, smaller 'shadow module' alongside each layer of the original LLM. This shadow module learns to refine the information flowing through each layer, working in parallel with the original model. Because these shadow modules are separate, they can be trained independently and even reused across different layers, making the process more efficient and flexible. It's like adding a helper to each layer instead of trying to adjust the layer itself.
Why it matters?
ShadowPEFT is important because it provides a competitive and potentially better way to adapt LLMs to new tasks without the huge cost of full retraining. It performs as well as or better than current methods while offering advantages like easier reuse of the adaptation modules and the possibility of running these modules on less powerful devices, like phones or edge computers, opening up new possibilities for where and how we can use these powerful AI models.
Abstract
Parameter-efficient fine-tuning (PEFT) reduces the training cost of full-parameter fine-tuning for large language models (LLMs) by training only a small set of task-specific parameters while freezing the pretrained backbone. However, existing approaches, such as Low-Rank Adaptation (LoRA), achieve adaptation by inserting independent low-rank perturbations directly to individual weights, resulting in a local parameterization of adaptation. We propose ShadowPEFT, a centralized PEFT framework that instead performs layer-level refinement through a depth-shared shadow module. At each transformer layer, ShadowPEFT maintains a parallel shadow state and evolves it repeatedly for progressively richer hidden states. This design shifts adaptation from distributed weight-space perturbations to a shared layer-space refinement process. Since the shadow module is decoupled from the backbone, it can be reused across depth, independently pretrained, and optionally deployed in a detached mode, benefiting edge computing scenarios. Experiments on generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.