QuatFuse: Quaternion-based orthogonal representation learning for multi-modal image fusion

Abstract

Multi-modality image fusion (MMIF) is a technique that integrates complementary information from different imaging modalities into a single image, aiming to generate a more comprehensive and information-rich integrated representation. The existing methods focus on using more complex network structures to improve the fusion performance of the model but ignore the correlation between different modal images. To solve this problem, we propose QuatFuse, a Quaternion-Based Orthogonal Representation Learning fusion method. This approach utilizes the mathematical properties of quaternions to model inter-modal relationships. Specifically, we introduce orthogonal geometric constraints and discrete cosine transformations to process redundant information and enhance features across various frequencies, effectively improving QuatFuse’s retention of key features. Fusing high-frequency and low-frequency information from multi-modal images after feature extraction is implemented in the quaternion domain, effectively mapping this processing procedure from the traditional real domain to a higher-dimensional representation space. To validate the robustness of QuatFuse, experiments on Infrared-Visible image fusion (IVF) and Medical image fusion (MIF) are conducted across 6 datasets (comprising 5 public datasets and 1 private dataset), with its performance being measured by eight distinct metrics. Our model achieved state-of-the-art (SOTA) performance on most evaluation metrics, demonstrating its superior fusion capabilities.