DP-SWAP: Fast Swapping Strategy Based on Dynamic Programming

Neural Architecture Search (NAS) has emerged as an effective approach for automating neural network design. However, NAS imposes significant GPU memory pressure due to the need to evaluate numerous candidate models during training. While tensor swapping helps reduce memory usage, existing tensor selection methods rely on extensive iterative searches, which require repeatedly traversing model computation graphs to evaluate the impact of swapping schemes–leading to high time complexity and poor scalability in dynamic NAS scenarios.

To address this issue, we propose DP-SWAP, a novel tensor swapping strategy based on dynamic programming. By leveraging the optimal substructure property of the tensor selection problem, DP-SWAP computes effective swapping schemes with only $O\left(n\right)$ time complexity, allows for fast and adaptive decision-making during NAS model exploration.

Experimental results show that DP-SWAP achieves training performance comparable to state-of-the-art heuristic methods, while reducing swapping decision time by over 3 orders of magnitude, thus effectively alleviating GPU memory bottlenecks in NAS.