Spatially Non-Stationary XL-MIMO Channel Estimation: A Three-Layer Generalized Approximate Message Passing Method

In this paper, the channel estimation problem for extremely large-scale multi-input multi-output (XL-MIMO) systems is investigated with the considerations of near-field (NF) spherical wavefront effects and spatially non-stationary (SnS) properties. Due to the diversity of SnS characteristics across different propagation paths, the concurrent channel estimation of multiple paths becomes intractable. To address this challenge, we propose a two-phase estimation scheme that decouples the problem into multiple subchannel estimation tasks. To solve these sub-tasks, we introduce a novel three-layer Bayesian inference scheme, exploiting the correlations and sparsity of the SnS subchannels in both the spatial and angular domains. Specifically, the first layer captures block sparsity in the angular domain, the second layer promotes SnS properties in the spatial domain, and the third layer effectively decouples each subchannel from the observed signal. To enable efficient Bayesian inference, we develop a three-layer generalized approximate message passing (TL-GAMP) algorithm that combines structured variational message passing with belief propagation rules. Simulation results validate the convergence and effectiveness of the proposed TL-GAMP algorithm, demonstrating its robustness across various channel environments, including NF-SnS, NF spatially stationary (NF-SS), and far-field spatially stationary (FF-SS) scenarios.