Compression-enhanced Three-Pass Protocol for secure and bandwidth-efficient image transmission

Abstract

Key-based cryptography faces persistent challenges in secure key distribution and newly rising vulnerabilities. Three-Pass Protocols (3PPs) tackle these issues through commutative encryption but typically triple bandwidth requirements. This paper addresses these bandwidth limitations by integrating an entropy-regularized Vector Quantized Variational Autoencoder (VQ-VAE) into a Fresnel-transform-based 3PP. Our VQ-VAE, trained on the Flickr8k dataset, achieves high-quality compression (average PSNR $\approx$ 31 dB, average MS-SSIM $\approx$ 0.96) at an average low bitrate ($<$ 0.35 bpp), reducing 3PP bandwidth requirements by over 97%. Comparative analysis at about 0.3 bpp demonstrates its competitive performance with recent state-of-the-art image compression techniques, and ablation studies validate the contribution of each key component to its overall efficacy. Compressed latent representations are then encrypted via commutative Fresnel transforms, enabling secure, keyless decryption. Security analysis reveals minimal correlation coefficient ($CC<0.04$) between original and encrypted latents, while decrypted latents fully recover the originals (CC $=1.0$). The final reconstructed images maintain high fidelity (CC $>0.98$). Furthermore, encrypted latents exhibit negligible adjacent-pixel correlation ($<$ 0.05), highlighting strong immunity to statistical attacks. Histogram analysis shows a high Kullback–Leibler (KL) divergence ($>$ 3.29) and a low histogram intersection (0.597) between ciphers and original latents, underscoring robust resistance to frequency-based methods. Sensitivity analysis reveals that a minute deviation in the diffraction distance ($10^{-5}$ m) severely degrades decryption quality (CC $<$ 0.05), demonstrating resistance against brute-force attacks. A lightweight verification step thwarts replay attacks, a known 3PP weakness. This framework enables secure and bandwidth-efficient image transmission, making it suitable for resource-constrained and security-critical applications such as telemedicine and remote-sensing downlinks.