Evaluation of Neural Demappers for Trainable Constellation in an End-to-End Communication System

Conventional M-ary Quadrature Amplitude Modulation (M-QAM) constellation designs such as rectangular constellation, are based on mathematical data and estimated channel models to achieve equal probability of error for transmitted bits in communication systems. However, these designs are suboptimal as they are not receptive to practical channel conditions or system performance due to their fixed lattice structure, and their performance degrades with a higher number of bits per symbol. Deep learning (DL) based end-to-end communication systems can be utilized to circumvent these challenges for better overall performance. Such systems are implemented as deep neural network (DNN) autoencoders, where trainable constellations and neural demappers (ND) can be jointly trained to achieve optimum constellation design for a higher data rate communication system. In this study, we evaluate the performance of two NDs that implement trainable constellation design and compared them with two baseline demapping algorithms in an end-to-end communication system. In the analysis, the NDs outperformed the baseline demappers for higher bits per symbol transmission, and trainable constellation design corresponding to 1024-QAM achieved the highest gain of 0.8 dB compared to the baseline demappers.