MemoryFusion: A novel architecture for infrared and visible image fusion based on memory unit

Abstract

Existing image fusion methods utilize elaborate encoders to sequentially extract shallow and deep features from the source images. However, most methods lack long-term dependence, i.e. shallow details are inevitably lost when the network encodes deep features. To this end, some methods employ skip connections or dense connections to directly assign shallow features into deeper layers, potentially introducing redundant information and increasing computational loads. To overcome these drawbacks and enhance the generalization ability for low-quality scenarios, a novel fusion architecture based on Gated Recurrent Unit (GRU) termed as MemoryFusion is proposed. First, the Input Extension Encoder (IEE) transfers the source image into a feature sequence. Then a Recurrent Fusion Encoder (RFE) containing Recurrent Memory Fusion Unit (RMFU) is designed to learn the intrinsic correlation between the multi-modality feature sequences and generate the fusion feature sequence. This memory fusion unit utilizes a special gating mechanism to incorporate historical information and current input, and then adaptively selects the valuable content and forgets the redundant information. More importantly, it effectively relieves the computational pressure. Finally, since the modality information is distributed at different sequence depths and varying illumination intensity, the Multi-hierarchical Aggregation Module (MHAM) is designed to obtain the corresponding weight sequence. The aggregated fusion feature is obtained by integrating the fusion feature sequence with the weight sequence. Extensive experiments demonstrate that MemoryFusion is superior to the state-of-the-art fusion methods on multiple datasets. Even on low-quality images, such as low-light or foggy conditions, our method also demonstrates exceptional fusion performance and scene fidelity.