Performance Analysis of Scheduling Algorithms in Apache Hadoop

Hadoop bundles the two computing resources of memory and CPU in the management resources, and then divides it into two resource models: MapSlot and ReduceSlot according to task types. MapReduce applications will have a large number of sorting operations in operation. Most of these sorts are executed iteratively, which consumes a lot of performance. Chapter 5 of this article takes this as an entry point and reorganizes the execution process of the Shuffle stage. Researched to replace quick sort with more efficient counting sorting. At the same time, the Shuffle execution is branched according to the definition of Combiner. One branch deletes the quick sort in the partition in the spill phase and the merge sort in the combine phase to reduce performance consumption. The other branch executes Combiner in advance to improve data processing efficiency. The two branches processed 21GB of log data on a 7-node PC cluster, and both achieved an efficiency improvement of about half an hour.