A temporally insensitive spatio-temporal fusion method for remote sensing imagery via semantic prior regularization

Abstract

Spatio-temporal fusion has become a popular technology for generating remote sensing images with high spatial and high temporal resolutions, thus providing valuable data support for remote sensing monitoring applications, such as environmental monitoring and city planning. Currently, deep learning-based methods have garnered a significant amount of attention, and they mostly employ the fine image at the neighboring date as an auxiliary image. However, capturing usable neighboring fine images may be challenging due to the adverse effects of weather conditions on optical images. Moreover, the fusion performance drops sharply when the temporal interval is long (i.e., there are significant differences in images). In this paper, we proposed a bidirectional pyramid fusion network with semantic prior regularization (BPFN-SPR), which exhibits remarkable flexibility and robustness to temporal intervals.

Specifically, the proposed BPFN-SPR contains dual-path operations (i.e., Semantic Extraction path and Image Reconstruction path). The semantic extraction path has two modes: parameter learning mode and parameter freezing mode. The parameter learning mode aims to learn the information representation of the auxiliary fine image, while the parameter freezing mode aims to perceive the accurate semantic information of the target fine image. The image reconstruction path progressively reconstructs spatial details of fine images from coarse images, which jointly optimizes the target fine image and the auxiliary fine image, reducing the temporal sensitivity of the reconstruction branch, and thereby improving its generalization ability. Experimental results show that the proposed method has competitive performance, especially for areas with land cover changes. In addition, extensive experiments using images at multi-temporal intervals as auxiliary images have also demonstrated the significant advantages of the proposed method. The mean PSNR value attains 31.0713, while the average spectral index SAM measures 0.1640 on the LGC test set. Meanwhile, for the CIA test set, the average PSNR is recorded at 29.5332, accompanied by an average spectral index SAM of 0.1865. Therefore, the proposed BPFN-SPR has considerable potential in monitoring Earth's surface dynamics.