Intelligent and efficient task-aligned nonlinear compensation for coherent optical fiber communication systems

Abstract

Fiber Kerr nonlinearity remains a critical bottleneck in high-capacity long-haul coherent optical communication systems, highlighting the importance of developing compensation schemes that balance performance and computational efficiency. Recently, neural network-based equalizers have attracted increasing attention due to their powerful nonlinear modeling capability. However, existing neural network-based methods often lead to undesired constellation deformation, compromising symbol integrity and limiting system reliability. We propose Task-Aligned center-oriented gated recurrent unit (Co-GRU), a low-complexity nonlinear compensation method that simultaneously addresses nonlinear distortion and constellation deformation through a task-coordinated strategy. In contrast to existing neural approaches that handle these objectives in isolation, our method integrates them into a unified optimization process, effectively decoupling compensation from geometric distortion and achieving accurate equalization. The method is experimentally assessed in both single-channel and wavelength-division multiplexing (WDM) systems, demonstrating its applicability across diverse transmission scenarios. In a single-channel experimental system operating at 64 Gbaud 16QAM over 2240 km standard single-mode fiber (SSMF), Task-Aligned Co-GRU achieves up to a 0.42 dB improvement in $Q^2$-factor over chromatic dispersion compensation (CDC), while outperforming one-step-per-span digital backpropagation (DBP-1) with a 75.5% reduction in computational complexity. In an eight-channel WDM experimental system transmitting 64 Gbaud 16QAM signals over 1600 km SSMF, it surpasses 16-step-per-span DBP (DBP-16) by 0.07 dB, requiring only 2.1% of its computational cost. These results demonstrate that Task-Aligned Co-GRU achieves a favorable trade-off between performance and complexity, supporting its potential for real-time deployment in high-capacity coherent optical networks.