{"title":"DP-SWAP: Fast Swapping Strategy Based on Dynamic Programming","authors":"Weiduo Chen , Xiaoshe Dong , Qiang Wang","doi":"10.1016/j.future.2025.108071","DOIUrl":null,"url":null,"abstract":"<div><div>Neural Architecture Search (NAS) has emerged as an effective approach for automating neural network design. However, NAS imposes significant GPU memory pressure due to the need to evaluate numerous candidate models during training. While tensor swapping helps reduce memory usage, existing tensor selection methods rely on extensive iterative searches, which require repeatedly traversing model computation graphs to evaluate the impact of swapping schemes–leading to high time complexity and poor scalability in dynamic NAS scenarios.</div><div>To address this issue, we propose DP-SWAP, a novel tensor swapping strategy based on dynamic programming. By leveraging the optimal substructure property of the tensor selection problem, DP-SWAP computes effective swapping schemes with only <span><math><mrow><mi>O</mi><mo>(</mo><mi>n</mi><mo>)</mo></mrow></math></span> time complexity, allows for fast and adaptive decision-making during NAS model exploration.</div><div>Experimental results show that DP-SWAP achieves training performance comparable to state-of-the-art heuristic methods, while reducing swapping decision time by over 3 orders of magnitude, thus effectively alleviating GPU memory bottlenecks in NAS.</div></div>","PeriodicalId":55132,"journal":{"name":"Future Generation Computer Systems-The International Journal of Escience","volume":"175 ","pages":"Article 108071"},"PeriodicalIF":6.2000,"publicationDate":"2025-08-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Future Generation Computer Systems-The International Journal of Escience","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0167739X25003656","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, THEORY & METHODS","Score":null,"Total":0}
引用次数: 0
Abstract
Neural Architecture Search (NAS) has emerged as an effective approach for automating neural network design. However, NAS imposes significant GPU memory pressure due to the need to evaluate numerous candidate models during training. While tensor swapping helps reduce memory usage, existing tensor selection methods rely on extensive iterative searches, which require repeatedly traversing model computation graphs to evaluate the impact of swapping schemes–leading to high time complexity and poor scalability in dynamic NAS scenarios.
To address this issue, we propose DP-SWAP, a novel tensor swapping strategy based on dynamic programming. By leveraging the optimal substructure property of the tensor selection problem, DP-SWAP computes effective swapping schemes with only time complexity, allows for fast and adaptive decision-making during NAS model exploration.
Experimental results show that DP-SWAP achieves training performance comparable to state-of-the-art heuristic methods, while reducing swapping decision time by over 3 orders of magnitude, thus effectively alleviating GPU memory bottlenecks in NAS.
期刊介绍:
Computing infrastructures and systems are constantly evolving, resulting in increasingly complex and collaborative scientific applications. To cope with these advancements, there is a growing need for collaborative tools that can effectively map, control, and execute these applications.
Furthermore, with the explosion of Big Data, there is a requirement for innovative methods and infrastructures to collect, analyze, and derive meaningful insights from the vast amount of data generated. This necessitates the integration of computational and storage capabilities, databases, sensors, and human collaboration.
Future Generation Computer Systems aims to pioneer advancements in distributed systems, collaborative environments, high-performance computing, and Big Data analytics. It strives to stay at the forefront of developments in grids, clouds, and the Internet of Things (IoT) to effectively address the challenges posed by these wide-area, fully distributed sensing and computing systems.