Evolving neural network for few-shot learning via training-free NAS

Efforts to improve the performance of Few-Shot Learning (FSL) have mainly centered around introducing more FSL approaches. However, the role of neural networks in FSL is less extensively analyzed. In this paper, we aim to bridge the gap between neural networks and FSL, and to propose a novel method from a training-free Neural Architecture Search (NAS) perspective to FSL. Specifically, we first conduct an in-depth analysis of Model Agnostic Meta Learning (MAML) based methods tailored to FSL, and find the main bottleneck of MAML-based methods for FSL that is attributed to the second-order of MAML. To address this issue, we introduce a new Theorem to ensure the first-order convergence of MAML. Then, we propose a novel Few-shot Neural Architecture Search (FNAS) framework to efficiently design neural architectures for FSL at initialization. FNAS introduces a training-free proxy by combining principles from Neural Tangent Kernel (NTK) and Fisher Information Matrix (FIM). This proxy effectively captures the expressivity of candidate architectures from the given search space in a training-free manner. To further enhance search efficiency, we integrate this proxy into an improved evolutionary algorithm to comprehensively explore the architecture space under minimal computational budgets. Experimental validation on mainstream benchmarks demonstrates that FNAS achieves state-of-the-art performance while being less costly in terms of computational budgets than its competitors.