CSC-DARTS: Efficient differentiable neural architecture search using channel splitting connections

Abstract

Recently, differentiable architecture search (DARTS) has made great progress in decreasing the computational cost of Neural Architecture Search (NAS). However, there is still a problem of excessive memory access costs in training the supernet. In this paper, we propose an efficient search framework for differentiable architecture search using channel splitting connections, namely CSC-DARTS, based on bi-level optimization and second-order gradient approximation. Specifically, a “Channel Splitting” technique is developed to split the feature maps of the supernet at the channel level into two branches, one of which is sent to the operation selection for less redundancy when exploring the search space, while bypassing the rest directly to the output as feature reuse. In addition, an “identity” connection is adopted in both the search and evaluation phases, which is regarded as a regularization for less variability, to bridge a large gap caused by the inconsistency of the architecture depth between the two stages. Experimental results on the benchmark datasets CIFAR-10, CIFAR-100, and ImageNet demonstrate that CSC-DARTS achieves state-of-the-art performance with fewer GPU resources, including a test error of 2.52 % on CIFAR-10, an average test error of 17.25 % on CIFAR-100, and a top-1/5 accuracy of 74.5 %/91.7 % on ImageNet.