Hybrid beamforming optimization design for millimeter-wave communications: A multi-scale convolutional neural network approach

Hybrid beamforming (HBF) particularly demonstrates outstanding performance in millimeter-wave (mmWave) massive MIMO communication systems due to its ability to enhance spectral efficiency while reducing hardware complexity and power consumption by combining the advantages of analog and digital beamforming. However, HBF design faces two major challenges. First, it heavily relies on precise channel state information (CSI) which is difficult to obtain in practical systems. Second, HBF design typically involves complex non-convex optimization problems due to the coupling of optimization variables between digital and analog beamforming. To address these issues, this paper proposes a general and high-performance multi-scale convolutional neural network (MSCNN) framework for optimizing HBF design. The network incorporating multi-scale convolution (MSC) module and channel attention mechanism (CAM) enhances feature extraction and optimization by leveraging estimated CSI inputs, which significantly improves processing capability and robustness. Considering the hardware limitation, we design a specialized Lambda layer that directly outputs the optimized analog beamforming vector to satisfy the constant modulus constraint. Additionally, an unsupervised deep learning strategy is adopted to train the neural network. By directly optimizing the negative spectral efficiency without relying on traditional training labels, the strategy reduces training overhead and improves model performance. Furthermore, the network is validated across multiple channel models, which demonstrates its excellent generalization in adapting to varying channel conditions. According to simulation results, the proposed MSCNN significantly outperforms existing methods in terms of spectral efficiency and convergence performance and exhibits compatibility with different channels.