Optimizing MapReduce efficiency and reducing complexity with enhanced particle Swarm Optimization (MR-MPSO)

Abstract

Big data's explosive growth poses serious data management difficulties, especially given the data's unequal distribution across massive databases. Because of this mismatch, traditional software systems are less effective, which leads to complex and wasteful data processing. We provide MapReduce-Modified Particle Swarm Optimization (MR-MPSO), a unique optimization technique, to tackle this problem. This strategy not only improves the administration of enormous datasets but also tackles the complexity issue of data imbalance. The MR framework is used to handle large-scale data processing tasks, with MR-MPSO driving the map and reducing functions. Our technique combines adaptive inertia weight with Particle Swarm Optimization (PSO) to improve the accuracy and efficiency of optimization for 10 unimodal and multimodal benchmark functions. MR-MPSO outperforms four optimization algorithms—MR K-means, MR bat, MR whale, and regular MR-on measures such as fitness value mean, median, and standard deviation. Furthermore, MR-MPSO regularly enhances throughput and average I/O rate, especially in complex write operations, with gains ranging from 1.4 % to 28.9 % in throughput and 2.1 % to 17.7 % in I/O rate over typical MR approaches.