Enhancing nonlinear compensation efficiency with multi-task neural networks for coherent optical systems.

Abstract

Traditional nonlinear compensation techniques often involve complex models that introduce significant computational overhead, particularly in high-speed, high-capacity optical communication systems. To address this challenge, we propose a low-complexity nonlinear compensation method based on a multi-task neural network (MT-NN), combined with a complexity-aware mean square error (MSE) and partial grid search optimization for coherent optical communication systems. The proposed framework exploits shared network weights to simultaneously process multiple symbols, thereby reducing redundant computations while maintaining compensation accuracy. Additionally, transfer learning (TL) is incorporated to further enhance training efficiency. Experimental results demonstrate that the MT-NN-based approach effectively lowers computational complexity across diverse optical transmission scenarios without compromising system performance. Compared to a conventional single-task neural network (ST-NN), our method achieves a superior trade-off between accuracy and efficiency. This work provides a promising solution for practical, low-complexity nonlinear compensation in next-generation optical communication systems.