A Job Dispatch Optimization Method on Cluster and Cloud for Large-Scale High-Throughput Computing Service

Abstract

Cloud technologies, clusters and grids have actively supported large-scale scientific computing over the years. Whereas these technologies provide unlimited computing resources, combining them with the existing infrastructures to effectively support demanding scientific applications is more and more laborious. In this paper, we design a service architecture and propose an algorithm to optimize job distribution on a cluster and a cloud using HTCaaS. HTCaaS is a pilot job-based multilevel scheduling system for large-scale scientific computing in Korea. In addition, we present a newly added cloud module on HTCaaS which is based on OpenStack. We implement and validate the algorithm in HTCaaS. A preliminary experiment is also conducted to find an optimal distribution ratio for CPU-intensive jobs and I/O-intensive jobs in our cloud and cluster environments. We compare our method to a baseline approach which distributes tasks in proportions of the number of cores each resource has in order to validate the proposed job dispatch optimization method. Experimental results show that the proposed method can improve throughput and match tasks to appropriate resources using adaptive job distribution ratio in cloud and cluster environments.