Efficient perturbation-aware distinguishing score for zero-shot neural architecture search

Abstract

Zero-cost proxies are under the spotlight of neural architecture search (NAS) lately, thanks to their low computational cost in predicting architecture performance in a training-free manner. The NASWOT score is one of the representative proxies that measures the architecture’s ability to distinguish inputs at the activation layers. However, obtaining such a score still requires considerable calculations on a large kernel matrix about input similarity. Moreover, the NASWOT score is relatively coarse-grained and provides a rough estimation of the architecture’s ability to distinguish general inputs. In this paper, to further reduce the computational complexity, we first propose a simplified NASWOT scoring term by relaxing its original matrix-based calculation into a vector-based one. More importantly, we develop a fine-grained perturbation-aware term to measure how well the architecture can distinguish between inputs and their perturbed counterparts. We propose a layer-wise score multiplication approach to combine these two scoring terms, deriving a new proxy, named efficient perturbation-aware distinguishing score (ePADS). Experiments on various NAS spaces and datasets show that ePADS consistently outperforms other zero-cost proxies in terms of both predictive reliability and efficiency. Particularly, ePADS achieves the highest ranking correlation among the advanced competitors (e.g., Kendall’s coefficient of 0.620 on NAS-Bench-201 with ImageNet-16-120 and 0.485 on NDS-ENAS), and ePADS-based random architecture search spends only 0.018 GPU days on DARTS-CIFAR to find networks with an average error rate of 2.64%.