A comprehensive survey on big data privacy and Hadoop security: Insights into encryption mechanisms and emerging trends

Abstract

Big data has transformed analytics and data processing in many different industries, but securing security and privacy in distributed systems like Hadoop is still rather complex. This article gives a deep analysis of the symmetric, asymmetric, and hybrid encryption techniques applied in Hadoop to preserve massive amounts of data. We critically analyze earlier research, underlining its advantages, flaws, and important trade-offs, specifically with reference to scalability, computing expense, and implementation complexity. Additionally, we analyze new improvements like blockchain integration and post-quantum encryption, analyzing their potential to increase Hadoop security. We find weaknesses in existing techniques via a comparative study and provide a hybrid encryption system aimed at secure and efficient data processing in Hadoop settings. Researchers and practitioners searching for scalable, privacy-preserving big data platform solutions should use this paper as a reference.