Maximum likelihood sequence estimation in high-speed PONs using machine learning-based pre-equalizers

High-speed passive optical networks (PONs) use advanced signal processing techniques like inter-symbol interference (ISI) equalization. While equalizers based on maximum likelihood sequence estimation (MLSE) via the Viterbi algorithm achieve excellent performance, they suffer from excessive implementation complexity except for very short channel responses. In this work, we employ a pre-equalizer for joint “channel shortening” and branch metric computation needed for the Viterbi algorithm. We then propose an optimization method for iteratively updating the pre-equalizer towards optimal end-to-end MLSE performance, by minimizing the multi-class cross-entropy loss based upon the path metrics. Numerical evaluations demonstrate that our proposed solution for MLSE with a small number of taps achieves significant ISI equalization improvements with respect to prior art approaches, and a performance close to MLSE with a high number of taps.