Development of pruning methods for a lightweight yet efficient AE-based communication system

Thanks to its superior bit error rate performance, the communication system based on autoencoder (AE) has recently attracted significant attention from both researchers and practitioners. In fact, communication functions, such as modulation and demodulation, are usually embedded in edge devices with constrained computing and memory resources. Consequently, the AE architecture within the communication system should be structurally optimized to reduce memory and computing requirements without sacrificing communication performance. Neural network pruning, a promising approach to complexity reduction, achieves this by removing redundant neurons or connections in neural networks. In this paper, we propose three novel pruning methods, such as weight pruning, neuron pruning, and hybrid pruning combining both, to reduce complexity in the AE-based communication system. In particular, the hybrid pruning method leverages the advantages of both weight and neuron pruning to further optimize the network architecture. Our evaluation results demonstrate the effectiveness of these pruning methods in model optimization, highlighting their potential applications in edge devices for modulation and demodulation tasks. Among the proposed pruning methods, the hybrid algorithm produces the best performance.