A Cooperative Flow Simulator for Distributed Computing Based on Full-Dimensional Definable Network

Yuan Liang, Geyang Xiao, Shaofeng Yao, Hongsheng Wang, Xiaoyu Yi, Yonggang Tu
{"title":"A Cooperative Flow Simulator for Distributed Computing Based on Full-Dimensional Definable Network","authors":"Yuan Liang, Geyang Xiao, Shaofeng Yao, Hongsheng Wang, Xiaoyu Yi, Yonggang Tu","doi":"10.1109/DOCS55193.2022.9967710","DOIUrl":null,"url":null,"abstract":"To improve the efficiency of cluster computing, the research of cooperative flow scheduling algorithms provides another idea for further optimization. Cluster computing frameworks such as Spark and Map-Reduce carry running cluster computing tasks and have distinct stage division and dependency characteristics. With limited network resources, the overall completion time of the job can be reduced by minimizing the wait time of the computing phase due to the bottleneck of transmission capacity. The introduction of an artificial intelligence algorithm can provide a solution to this NP-hard problem. The simulator designed in this paper can realize the overall process of cooperative traffic generation, scheduling policy configuration, and scheduling effect feedback closed-loop through packet sender, packet receiver, scheduler, and programmable switch components according to the dependency between cluster computing stages.","PeriodicalId":348545,"journal":{"name":"2022 4th International Conference on Data-driven Optimization of Complex Systems (DOCS)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2022-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2022 4th International Conference on Data-driven Optimization of Complex Systems (DOCS)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/DOCS55193.2022.9967710","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

To improve the efficiency of cluster computing, the research of cooperative flow scheduling algorithms provides another idea for further optimization. Cluster computing frameworks such as Spark and Map-Reduce carry running cluster computing tasks and have distinct stage division and dependency characteristics. With limited network resources, the overall completion time of the job can be reduced by minimizing the wait time of the computing phase due to the bottleneck of transmission capacity. The introduction of an artificial intelligence algorithm can provide a solution to this NP-hard problem. The simulator designed in this paper can realize the overall process of cooperative traffic generation, scheduling policy configuration, and scheduling effect feedback closed-loop through packet sender, packet receiver, scheduler, and programmable switch components according to the dependency between cluster computing stages.
基于全维可定义网络的分布式计算协同流模拟器
为了提高集群计算的效率,协同流调度算法的研究为进一步优化提供了另一种思路。Spark、Map-Reduce等集群计算框架承载运行集群计算任务,具有明显的阶段划分和依赖特征。在网络资源有限的情况下,通过最小化由于传输容量瓶颈导致的计算阶段的等待时间,可以减少作业的总体完成时间。人工智能算法的引入可以为这个np困难问题提供一个解决方案。本文设计的仿真器可以根据集群计算阶段之间的依赖关系,通过分组发送者、分组接收者、调度器和可编程交换机等组件,实现协同流量生成、调度策略配置和调度效果反馈闭环的全过程。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信