Multilinear hyperspectral unmixing based on autoencoder and recurrent neural network

Spectral unmixing techniques estimate the endmember spectra and corresponding abundance fractions that constitute the pixels of hyperspectral remote sensing images, revealing the mixing mechanisms of materials within pixels. In recent years, deep learning has shown significant potential in advancing spectral unmixing, particularly in nonlinear scenarios. However, most existing nonlinear models rely on bilinear mixing frameworks, with limited focus on high-order nonlinear models. This restricts their ability to capture complex interactions such as multiple light scattering events. To address this issue, this work proposes an unsupervised unmixing method leveraging an autoencoder network framework and the multilinear mixing model (MLM). It employs a recurrent neural network (RNN) in the decoder to simulate the multiple scattering of light between materials. Unlike conventional multilinear approaches that rely on explicit mathematical formulations, the proposed method leverages the RNN to automatically learn and approximate the nonlinear interactions of light. Moreover, the RNN weights are adaptively updated during training and interpreted as transition probabilities representing further light interactions among materials, endowing the model structure with explicit physical interpretation. Besides, a new stopping criterion is also designed, which ensures better RNN weights are obtained during backpropagation. Experiments conducted on both synthetic and real datasets demonstrate the better performance of the proposed method.