Surrogate-Assisted Niching Differential Evolution for hyperparameter optimization in Convolutional Neural Networks

Abstract

Hyperparameter optimization in deep Convolutional Neural Networks (CNNs) plays a crucial role in enhancing model performance. However, such problems often exhibit inherent challenges, including high-dimensionality, multimodality, expensive evaluations, and mixed-variable characteristic, which impose high demands on optimization algorithms. To address these challenges, this study proposes a Surrogate-Assisted Niching Differential Evolution (SANDE), which efficiently optimizes CNN architecture hyperparameters through landscape delineation, surrogate-assisted optimization, and computational resource allocation. Specifically, SANDE employs a niching technique that integrates information from both the decision and objective spaces to divide the hyperparameter landscape into multiple simpler and promising sub-regions. These sub-regions are then efficiently searched using a surrogate-assisted integrated differential evolution, where a hybrid differential evolution and a surrogate model serve as the optimizer and objective function, respectively. An information integration strategy is also incorporated to enhance the optimization robustness and convergence. Furthermore, a dynamic resource allocation strategy is introduced to distribute computational resources across sub-regions based on their accumulated historical optimization outcomes, thereby enhancing resource utilization efficiency. The resulting SANDE-CNN is compared with six manually designed CNNs and 23 CNNs obtained by 14 advanced CNN optimization methods. Experimental results demonstrate that SANDE can achieve competitive performance at the cost of limited computational resources.