Performance Evaluation of Apache Hadoop Benchmarks under a Dynamic Checkpointing Mechanism

Abstract

The increasingly robust High Performance Computing (HPC) systems have promoted many researches on fault tolerance mechanisms. However, the efficiency of these techniques is an important challenge, as the data and resources demand is also getting higher. in this work, we presents how the framework Apache Hadoop implements the Checkpoint and Recovery technique for fault tolerance providing on its distributed file system (Hadoop Distributed File System). The demand of large amounts of data from all applications supported by Hadoop framework can compromise the mechanism efficiency, once its configurations attributes are defined statically. So we present a dynamic configuration mechanism to checkpoint for Apache Hadoop through a resource monitoring, whose goal is to make the Hadoop checkpoint adaptable. The proposed mechanism is then evaluated and submitted to performance tests.