SecureMD5: A new stream cipher for secure file systems and encryption key generation with artificial intelligence

The insider threat to sensitive information posed by employees or partners of an organisation remains a major cybersecurity challenge. In this regard, the measures taken by organisations and companies to protect information are often insufficient. Primarily, due to the legitimate access and knowledge of security holes that these individuals possess.

This study proposes SecureMD5, an encryption algorithm designed specifically for secure file systems (SFS). The algorithm is based on custom one-way functions integrated into an encryption scheme that operates at the byte level. It uses 11 dynamic variables generated from contextual parameters such as file position, access time, random values, and user-specific keys. This approach ensures that SecureMD5 does not inherit the known vulnerabilities of MD5 as a standard cryptographic algorithm. Consequently, SecureMD5 is presented as an adaptive and robust solution that addresses the challenges posed by insider threats in SFS.

In parallel, a modular contextual key generation scheme is proposed, which can incorporate various challenges such as user identity, access time and device location. Biometric key generation based on Artificial Intelligence (AI) methods is evaluated independently from the validation of the encryption algorithm. In the evaluated biometric key generation scheme, the AI models MediaPipe Hand Landmark and LBPHFaceRecognizer from OpenCV have been used. These methods are part of a sub-key generation scheme based on contextual challenges. This scheme eliminates the need for key storage for dynamic and secure access to sensitive information.

SecureMD5 was validated by diffusion, confusion, entropy and performance analysis. It achieved 31 % higher entropy than comparable algorithms. Performance improved by 0.32 % compared to RC4. It also passed 87 % of NIST 800–22 tests, demonstrating its robustness against cryptographic vulnerabilities. In addition, SecureMD5 balances security and performance, with encryption times 25 % faster than a modified AES algorithm for 10 MB files. Biometric key generation methods were evaluated using metrics such as precision, accuracy, false acceptance rate and specificity, achieving satisfactory values above 80 % on all metrics. This work addresses critical gaps in information security, providing significant advances in protecting SFS against insider threats. The design and adaptability of SecureMD5 make it particularly suitable for sectors with strict security requirements, such as healthcare, finance, and corporate data management. Its ability to enable dynamic and secure access control addresses the real challenges posed by protecting confidential information from internal threats.