Accurate mode classification and image transmission via optimized deformable triple graph convolution network in fractional spatial mode optical communication

Free-space optical (FSO) communication systems leveraging spatial-mode multiplexing offer high-capacity, low-interference data transmission, but existing methods often struggle with accurately classifying closely spaced fractional spatial modes due to limitations in feature representation, neighborhood modeling, and noise sensitivity. To overcome these challenges, this work proposes a novel Deformable Triple Attention Graph Convolutional Network with Dollmaker Optimization Algorithm (DTAGCN-DOA) for robust mode classification and high-resolution recognition of fractional Bessel-Gaussian (BG) beam-encoded data. Fractional BG beams are generated using a He–Ne laser modulated via a Spatial Light Modulator (SLM), encoding 8-bit digital symbols into structured spatial modes. A total of 51,200 beam intensity images, each corresponding to unique combinations of radial wave number and fractional orbital angular momentum, are collected and preprocessed using Masked Joint Bilateral Filtering (MJBF) to suppress noise while preserving fine spatial features. Feature extraction is conducted via a Boundary-Enhanced Patch-Merging Transformer (BEPMT), combining global semantics and boundary details. The proposed DTAGCN-DOA framework classifies fractional spatial modes using deformable graph convolutions and triple attention mechanisms, with hyperparameters optimized via the DOA. Grayscale images are encoded into 15,000 spatial modes, transmitted optically, and decoded using the trained model. It achieves 99.88 % accuracy, 31.4 dB PSNR, and 0.995 confidence, ensuring reliable, high-fidelity FSO data recognition.