Deep Learning-Based Two-Step Denoising and Detection for MIMO-NOMA Under Imperfect Conditions

Non-orthogonal multiple access (NOMA) has the potential to pave the way for next-generation wireless systems, especially when combined with multiple-input multiple-output (MIMO) technology. However, conventional detection schemes such as successive interference cancellation (SIC) often suffer from performance degradation under imperfect conditions. This letter presents a novel two-step deep learning (DL) framework for MIMO-NOMA signal detection. The proposed two-step deep learning (DL) framework first employs an autoencoder (AE) to denoise the received signal, and then feeds it to the deep neural network (DNN) for signal detection. The proposed architecture utilizes an AE with convolutional layers and a dual-receiver DNN signal detection for each user signal, eliminating the need for SIC. Simulation results demonstrate that the proposed system achieves a 10 dB bit error rate (BER) gain at 20 dB SNR and an achievable rate exceeding 12 bps/Hz, outperforms other DL-based detection methods, and conventional SIC.