LPZero: Language Model Zero-cost Proxy Search from Zero
Peijie Dong, Lujun Li, Xiang Liu, Zhenheng Tang, Xuebo Liu, Qiang Wang, Xiaowen Chu
2024-10-13
Summary
This paper discusses LPZero, a new framework designed to automatically create zero-cost proxies for optimizing neural architecture search (NAS) in natural language processing (NLP) tasks.
What's the problem?
Neural Architecture Search (NAS) is a method used to find the best model architecture for machine learning tasks, but it requires a lot of computational resources, which can be very expensive and time-consuming. Existing zero-cost proxies, which are supposed to help reduce these costs, often depend heavily on expert knowledge and involve a lot of trial and error, making them inefficient and sometimes ineffective.
What's the solution?
To tackle these issues, the authors introduce LPZero, which automatically designs zero-cost proxies without needing expert input. They treat these proxies as symbolic equations and utilize genetic programming to search for the best combinations of mathematical symbols that can represent the proxies effectively. Additionally, they implement a Rule-based Pruning Strategy (RPS) to eliminate less promising proxies early in the process. This approach allows LPZero to outperform existing methods in terms of ranking consistency and overall performance on various NLP tasks.
Why it matters?
This research is important because it simplifies the process of optimizing neural architectures by reducing the reliance on expert knowledge and minimizing computational costs. By improving how zero-cost proxies are designed, LPZero could make it easier and faster to develop effective machine learning models, ultimately advancing the field of artificial intelligence.
Abstract
In spite of the outstanding performance, Neural Architecture Search (NAS) is criticized for massive computation. Recently, Zero-shot NAS has emerged as a promising approach by exploiting Zero-cost (ZC) proxies, which markedly reduce computational demands. Despite this, existing ZC proxies heavily rely on expert knowledge and incur significant trial-and-error costs. Particularly in NLP tasks, most existing ZC proxies fail to surpass the performance of the naive baseline. To address these challenges, we introduce a novel framework, LPZero, which is the first to automatically design ZC proxies for various tasks, achieving higher ranking consistency than human-designed proxies. Specifically, we model the ZC proxy as a symbolic equation and incorporate a unified proxy search space that encompasses existing ZC proxies, which are composed of a predefined set of mathematical symbols. To heuristically search for the best ZC proxy, LPZero incorporates genetic programming to find the optimal symbolic composition. We propose a Rule-based Pruning Strategy (RPS), which preemptively eliminates unpromising proxies, thereby mitigating the risk of proxy degradation. Extensive experiments on FlexiBERT, GPT-2, and LLaMA-7B demonstrate LPZero's superior ranking ability and performance on downstream tasks compared to current approaches.