An approach for optimization of resource management in Hadoop

Abstract

Many tools and frameworks have been developed to process data on distributed data centers. MapReduce most prominent among such frameworks has emerged as a popular distributed data processing model for processing vast amount of data in parallel on large clusters of commodity machines. The JobTracker in MapReduce framework is responsible for both managing the cluster's resources and executing the MapReduce jobs, a constraint that limits scalability, resource utilization. YARN the next-generation execution layer for Hadoop splits processing and resource management capabilities of JobTracker into separate entities and eliminates the dependency of Hadoop on MapReduce. This new model is more isolated and scalable compared to MapReduce, providing improved features and functionality. This paper discusses the design of YARN and significant advantages over traditional MapReduce.