Towards robust multimodal fault diagnosis of electromechanical systems with limited labeled data via cross-modal self-contrastive learning

Multimodal signals with comprehensive and complementary information have been successfully applied in fault diagnosis of electromechanical systems. However, the scarcity of labeled multimodal signal data, coupled with inevitable distribution shifts, poses a significant challenge to the effective training of multimodal diagnostic models. Moreover, the availability of all modalities cannot always be guaranteed during extended inference periods, which can further induce significant performance degradation. Therefore, this paper proposes a robust multimodal fault diagnosis method with limited labeled data via a cross-modal self-contrastive learning (CMSCL) model. First, heterogeneous multimodal signals are collected and preprocessed to extract unified fault characteristics across different modalities. Then, the CMSCL model is initially pre-trained through modality-masking self-supervised learning on unlabeled signal data and subsequently fine-tuned with limited labeled data for fault diagnosis. Finally, a missing modalities completion module is designed and integrated into the CMSCL model to further address the missing modality issue. Extensive experimental results on two public experimental rig datasets and a real-world industrial dataset from different electromechanical systems demonstrate the superior accuracy and robustness of the proposed method compared with state-of-the-art approaches.