STMFuse: spatiotemporal feature saliency change-driven mimic fusion for infrared and visible video

Abstract

The variable and unpredictable distribution of spatiotemporal features in infrared–visible videos under complex dynamic environments is addressed, where traditional fusion algorithms with fixed architectures are found to fail when adapting to continuous dynamic changes in feature distributions, resulting in blurred fusion outcomes and loss of critical detail information. To overcome this bottleneck, a mimic fusion method based on spatiotemporal feature saliency change-driven approach, termed STMFuse, is proposed, whereby spatiotemporal feature variations are continuously monitored and the fusion architecture is dynamically reconfigured, thereby ensuring optimal fusion performance. Specifically, a spatiotemporal dual-domain feature perception module extracts single-modal temporal change features and cross-modal difference features to comprehensively capture dynamic scene characteristics. To address the inadequacy of single threshold settings in measuring dynamic feature variations, a possibility distribution function and synthesis rules is constructed to quantify feature change degree, utilizing significant change features as driving factors for mimic variant adjustments. By investigating the dynamic correlation between feature changes and fusion quality metrics, a mimic variant evaluation function based on correlation coefficient weighting is established, mapping the relationship between features and mimic variants. Finally, a multi-feature collaborative decision mechanism based on energy functions is designed to determine the optimal mimic variant combination by integrating various feature change characteristics. Extensive experimental validation demonstrates that STMFuse significantly outperforms existing methods in terms of adaptive fusion performance and robustness in dynamic scenes, exhibiting superior preservation of edges, texture details, and enhanced visual perception quality.