Optical fiber nonlinear neural compensation based on generalized mutual information cost function

Abstract

Optical fiber nonlinearity is a core limitation to high-capacity transmission in fiber communication. Data-driven methods for fiber nonlinear compensation, utilizing neural networks combined with digital signal processing technology, have shown remarkable performance. However, many of these neural networks are trained using the mean square error (MSE) cost function, which can lead to a non-Gaussian distribution after compensation. Although the bit error rate improves with compensation, the non-Gaussian distribution is not well-suited for forward error correction (FEC) and can result in performance degradation after soft decoding. In this paper, we propose a general mutual information (GMI) cost function based on Gaussian distribution constraints, which addresses the performance loss associated with MSE after FEC. In an additive Gaussian white noise channel, where no nonlinearity exists, the proposed GMI cost function achieves equivalent performance after FEC decoding, whereas the MSE cost function results in more than 1.0 dB signal-to-noise ratio loss. Combined with a long short-term memory network, 1600 km transmission experiments show that the MSE cost function leads to a 0.45 dB loss after FEC decoding, while the GMI cost function provides a 0.8 dB improvement over the MSE cost function. The proposed GMI cost function sets a new training criterion for nonlinear compensation in optical fiber communications.