S-box using fractional-order 4D hyperchaotic system and its application to RSA cryptosystem-based color image encryption

An algorithm for substitution box (S-box) construction is proposed by using fractional-order (FO) 4D hyperchaotic system in this paper. Based on hyperchaotic sequences generated by the FO 4D hyperchaotic system, exclusive-or (XOR) operations, and permutation the S-box is constructed. A comprehensive evaluation based on various algebraic criteria demonstrates that the proposed S-box exhibits favorable performance in terms of strict avalanche criterion (SAC), differential approximation probability (DP), nonlinearity (NL), bit independence criterion (BIC), linear approximation probability (LP), fixed point (FP), reverse fixed point (FPr), and correlation immunity (CI). The proposed S-box exhibits excellent NL of 112, comparable to that of the Advanced Encryption Standard (AES) S-box. Furthermore, a novel color image encryption technique is developed to ensure the privacy of color images. The proposed color image encryption algorithm combines Arnold Cat map (ACM), a high-performance generated S-box, Rivest–Shamir–Adleman (RSA) algorithm with 2D Henon chaotic map (HCM). Through rigorous testing, analysis, and comparison with recently reported image encryption algorithms, the suggested algorithm demonstrates its security, efficiency, and resistance to different standard attacks. Notably, the suggested algorithm for color image encryption offers a substantial key space and achieves an approximate correlation of 0 between adjacent pixels. Additionally, the entropy approaches the ideal value of 8.