Sparse DARTS with Various Recovery Algorithms

Abstract

Designing an efficient neural architecture search method is an open and challenging problem over the last few years. A typical and well-performed strategy is gradient-based methods (i.e., Differentiable Architecture Search (DARTS)), which mainly searches the target sparse child graph from a trainable dense super graph. However, during the searching phrase, training the dense super graph usually requires excessively computational resources. Besides, the training based on a dense graph is excessively inefficient, and the memory consumption is prohibitively high. To alleviate this shortcoming, recently Iterative Shrinkage Thresholding Algorithm (ISTA), a sparse coding recovery algorithm, has been applied to DARTS, which directly optimizes the compressed representation of the super graph, and saves the memory and time consumption. Indeed, there are several such kinds of sparse coding recovery algorithms, and ISTA is not the best one in terms of recovery efficiency and effectiveness. To investigate the impact of different sparse coding recovery algorithm on performance in DARTS and provide some insights. Firstly, we design several sparse DARTS based on different sparse coding recovery algorithms (i.e., LISTA, CoD, and Lars). Then a series of controlled experiments on selected algorithms are conducted. The accuracy, search time and other indicators of the model are collected and compared. Sufficient theoretical analysis and experimental exploration reveal that the different compression algorithms show different characteristics on the sparse DARTS. Specifically, Lars-NAS tends to choose the operation with fewer parameters, while Cod-NAS is the simplest of the four recovery algorithms, and its consuming time is very short, but the CoD-NAS model is unstable. Particularly, LISTA-NAS achieves the accurate results with stable recovery time. Thus, it can be seen that all compression algorithms are available to utilized according to different environments and requirements.