The impact of checkpointing interval selection on the scheduling performance of Hadoop framework

Abstract

MapReduce is one of the most popular paradigm for processing a huge volume of data (big data) in distributed manner. In addition, Hadoop is considred as one of the most well-known implemention of MapReduce for processing MapReduce programs. The scheduler in Hadoop manages and monitors the scheduling of tasks. In addition, if a failure takes place, Hadoop reschedules the failed tasks. This makes fault tolerance a critical issue for the efficient operation of any application running on Hadoop in order to ensure the quality of service (QoS) and to meet the end-users expectations. Among the well-used techniques for providing fault tolerance in distributed systems, there is the checkpointing technique. The idea behind checkpointing for MapReduce tasks is to use checkpoints to save intermediate results at some points in time. Once a task fails, it can restart from the checkpointed state. However, selecting an appropriate checkpointing interval is not a trivial task. Unnecessary frequent checkpointing may degrade the system performance. Consequently, the checkpointing interval must be selected taking into account the failure probability, as well as the nature of the workload. Towards this direction, we have analyzed the performance of Hadoop with presence of different types of failures (task failure, TaskTracker failure and NameNode failure). We then investigate via simulation the impact of checkpointing interval selection on the performance of Hadoop under various failure probabilities. This paper also discusses our findings and draws attention on how to improve the checkpointing interval selection on Hadoop.