GAPrune: Gradient-Alignment Pruning for Domain-Aware Embeddings
Yixuan Tang, Yi Yang
2025-09-16
Summary
This paper focuses on making powerful AI models for specific tasks, like understanding code or financial information, smaller and easier to use without losing their accuracy.
What's the problem?
Current AI models that are good at specialized tasks are often huge, needing lots of computing power to run. Simply shrinking these models doesn't work well because it's hard to tell which parts of the model are important for the specific task and which parts represent general knowledge. Existing methods treat all parts of the model equally when shrinking it, which leads to a loss of performance on the specialized task.
What's the solution?
The researchers developed a new method called GAPrune that intelligently shrinks these models. It figures out which parts of the model are most important for the specific task *and* which parts are crucial for general language understanding. It then prioritizes keeping those important parts while removing the less important ones. They use a scoring system, called Domain Alignment Importance, that combines how important a parameter is for the specific task with how much it contributes to general language skills. This allows for a more targeted and effective shrinking process.
Why it matters?
This work is important because it allows for the creation of specialized AI models that can run on devices with limited resources, like phones or embedded systems. By preserving and even improving performance on specific tasks while reducing model size, GAPrune opens up possibilities for deploying these powerful AI tools in a wider range of applications and makes them more accessible.
Abstract
Domain-specific embedding models have shown promise for applications that require specialized semantic understanding, such as coding agents and financial retrieval systems, often achieving higher performance gains than general models. However, state-of-the-art embedding models are typically based on LLMs, which contain billions of parameters, making deployment challenging in resource-constrained environments. Model compression through pruning offers a promising solution, but existing pruning methods treat all parameters uniformly, failing to distinguish between general semantic representations and domain-specific patterns, leading to suboptimal pruning decisions. Thus, we propose GAPrune, a pruning framework that addresses this challenge by considering both domain importance and preserving general linguistic foundation. Our method uses Fisher Information to measure importance and general-domain gradient alignment to assess parameter behavior, then combines these signals using our Domain Alignment Importance (DAI) scoring. Lower DAI scores indicate that the parameter is either less important for the domain task or creates conflicts between domain and general objectives. Experiments on two domain benchmarks, FinMTEB and ChemTEB, show that GAPrune maintains performance within 2.5% of dense models in one-shot pruning at 50% sparsity, while outperforming all baselines. With retraining in 100 steps, GAPrune achieves +4.51% improvement on FinMTEB and +1.73% on ChemTEB, demonstrating that our pruning strategy not only preserves but enhances domain-specific capabilities. Our findings demonstrate that principled pruning strategies can achieve model compression and enhanced domain specialization, providing the research community with a new approach for development.