An Optimized Job Scheduling Mechanism for Mapreduce Framework Using DIW-WOA in Big Data

Abstract

For processing, storing, and managing huge amounts of data, big data has turned out to be famous. The most popular map reduce programming model is the open-source Hadoop. It executes the MapReduce (MR) programming model to process big data with MR jobs. Nevertheless, scheduling MR jobs across multiple nodes has been regarded as an optimization issue in spite of current endeavors toward ameliorating MR's performance. Hence, to carry out an MR Job scheduling centered on Completion Time (CT) and cost whilst meeting the security constraints, a new optimization algorithm like Decreasing Inertia Weight induced Whale Optimization Algorithm (DIW-WOA) is proposed. At first, for big data clustering, a Butterfly Optimization-centered K-Means clustering Algorithm (BO-KMA) is applied. At last, for scheduling the client's tasks or jobs to the optimized scheduler. The whale optimization algorithm is hybridized with inertia weight concept and finally DIW-WOA is employed. When weighed against the other well-known algorithms, the experimental outcomes exhibit that the proposed algorithms for data clustering, DS and Task Scheduling (TS) achieve better performance in terms of make-span and throughput. It can be incorporated into the Hadoop environment.