Asymmetric simulation-enhanced flow reconstruction for incomplete multimodal learning

Abstract

Incomplete multimodal learning addresses the common real-world challenge of missing modalities, which undermines the performance of standard multimodal methods. Existing solutions struggle with distribution mismatches between reconstructed and observed data, asymmetric cross-modal structures, and insufficient cross-modal knowledge sharing. To tackle these issues, we propose an asymmetric simulation-enhanced flow reconstruction (ASE-FR) framework, which contains following contributions: (1) Distribution-consistent flow reconstruction module that align available and missing modality distributions by normalizing flows; (2) Asymmetric simulation module that perturbs and randomly masks features to mimic real-world modality absence and improve robustness; (3) Modal-shared knowledge distillation that transfers shared representations from teacher encoders to a student encoder through contrastive learning. This framework is applicable to a range of real-world scenarios, such as multi-sensor networks in smart manufacturing, medical diagnostic systems combining imaging and electronic health records, and autonomous driving platforms that integrate camera and LiDAR data. The experimental results show that our ASE-FR method achieves 94.71 %, 41.85 % and 81.90 % accuracy on Audiovision-MNIST, MM-IMDb and IEMOCAP datasets, as well as 1.1376 error rate on CMU-MOSI dataset, which exhibits competitive performance.