A Cooperative Flow Simulator for Distributed Computing Based on Full-Dimensional Definable Network

Abstract

To improve the efficiency of cluster computing, the research of cooperative flow scheduling algorithms provides another idea for further optimization. Cluster computing frameworks such as Spark and Map-Reduce carry running cluster computing tasks and have distinct stage division and dependency characteristics. With limited network resources, the overall completion time of the job can be reduced by minimizing the wait time of the computing phase due to the bottleneck of transmission capacity. The introduction of an artificial intelligence algorithm can provide a solution to this NP-hard problem. The simulator designed in this paper can realize the overall process of cooperative traffic generation, scheduling policy configuration, and scheduling effect feedback closed-loop through packet sender, packet receiver, scheduler, and programmable switch components according to the dependency between cluster computing stages.