Realizing Prioritized Scheduling Service in the Hadoop System

Abstract

Cloud computing has been widely used in many areas nowadays. It is common that large cloud systems could simultaneously service tens of thousands of users and host an excessive number of jobs running at the same time. Under such circumstances, the completion of urgent or time-critical tasks can be significantly delayed if the underlying cloud system does not offer schemes to speed up the execution of those tasks. Among the platforms adopted in cloud computing, Hadoop is one of the most widely used in the community of cloud computing. Unfortunately, Hadoop does not provide users efficient ways to expedite the course of execution for high-priority jobs which users would hope for their fast completion. We designed and implemented a new scheduling scheme enabling Hadoop to support fully prioritized scheduling. With our scheduling scheme, users can dynamically assign high priority to individual jobs so their execution could be accelerated accordingly. We evaluated our design and implementation by executing the same programs with ordinary priority versus high priority in Hadoop environments under different configurations. Experimental results show that programs can shorten their execution time by up to 82.70% if they are executed with high priority.