Dynamic deployment model for large-scale compute-intensive clusters*

Abstract

In big data computing application, there are jobs of multiple computing modes mainly based on MapReduce. Therefore, compute-intensive cluster needs to maintain multiple computing modes. The utilization of virtual computing resources is not high because of the change of computing load. In order to optimize the resource utilization of virtual cluster, a dynamic deployment model is designed with the support of lightweight Docker container technology. The deployment model can adjust the form of virtual cluster according to the change of the resource request of job, mainly changing the type and size of computing nodes in real time. Simulation experiments show that the dynamic deployment model can optimize the utilization of virtual resources, with CPU utilization increased by 5.2%, and the execution efficiency of computing jobs is optimized. The dynamic deployment model can be applied to cloud environment and large-scale computing clusters, not only to achieve peak-taggering computing of user jobs, but also to achieve the purpose of dynamic customization of job execution environment.