{"title":"A Cooperative Flow Simulator for Distributed Computing Based on Full-Dimensional Definable Network","authors":"Yuan Liang, Geyang Xiao, Shaofeng Yao, Hongsheng Wang, Xiaoyu Yi, Yonggang Tu","doi":"10.1109/DOCS55193.2022.9967710","DOIUrl":null,"url":null,"abstract":"To improve the efficiency of cluster computing, the research of cooperative flow scheduling algorithms provides another idea for further optimization. Cluster computing frameworks such as Spark and Map-Reduce carry running cluster computing tasks and have distinct stage division and dependency characteristics. With limited network resources, the overall completion time of the job can be reduced by minimizing the wait time of the computing phase due to the bottleneck of transmission capacity. The introduction of an artificial intelligence algorithm can provide a solution to this NP-hard problem. The simulator designed in this paper can realize the overall process of cooperative traffic generation, scheduling policy configuration, and scheduling effect feedback closed-loop through packet sender, packet receiver, scheduler, and programmable switch components according to the dependency between cluster computing stages.","PeriodicalId":348545,"journal":{"name":"2022 4th International Conference on Data-driven Optimization of Complex Systems (DOCS)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2022-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2022 4th International Conference on Data-driven Optimization of Complex Systems (DOCS)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/DOCS55193.2022.9967710","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
To improve the efficiency of cluster computing, the research of cooperative flow scheduling algorithms provides another idea for further optimization. Cluster computing frameworks such as Spark and Map-Reduce carry running cluster computing tasks and have distinct stage division and dependency characteristics. With limited network resources, the overall completion time of the job can be reduced by minimizing the wait time of the computing phase due to the bottleneck of transmission capacity. The introduction of an artificial intelligence algorithm can provide a solution to this NP-hard problem. The simulator designed in this paper can realize the overall process of cooperative traffic generation, scheduling policy configuration, and scheduling effect feedback closed-loop through packet sender, packet receiver, scheduler, and programmable switch components according to the dependency between cluster computing stages.