PROVING THE SECURITY OF AES BLOCK CIPHER BASED ON MODIFIED MIXCOLUMN

Block ciphers in general, Substitution-Permutation Network (SPN) block ciphers in particular are cryptographic fields widely applied today. AES is an SPN block cipher used in many security applications. However, there are many strong attacks on block ciphers as linear attacks, differential attacks, and algebraic attacks which are challenging for cryptographers. Therefore, the research to improve the security of block ciphers in general and AES, in particular, is a topic of great interest today. Along with security, the issue of the execution cost of block ciphers is also crucial in practice. In this paper, we clarify the role of the MDS matrix in increasing the branch number of the diffusion layer of the block ciphers, thereby improving the security of the block ciphers. We propose a method improving the security of the AES block cipher by changing the Mixcolumn transformation of AES using execution-efficient MDS matrices of size 4, 8, or 16. We present a method to find a new diffusion matrix of modified AES block ciphers from which to evaluate the number of fixed points and coefficient of fixed points  of the modified AES diffusion layers. In addition, we prove the branch number of the modified AES diffusion layers with MDS matrices of sizes 8, and 16. Then we also analyze the security, statistical standards and execution speed of modified AES block ciphers generated from those MDS matrices. The results show that our proposed method can significantly improve the security of the AES block cipher.