T2MFDF: An LLM-Enhanced Multimodal Fault Diagnosis Framework Integrating Time-Series and Textual Data

In modern industrial applications, accurate fault diagnosis is critical for ensuring machinery reliability, yet traditional methods struggle with the complexity and interdependencies of faults, particularly in bearing systems. This article proposes a novel multimodal fault diagnosis framework that integrates time-series vibration signals with textual descriptions, leveraging a BERT-based large language model (LLM) to enhance feature representation and capture semantic relationships between fault categories. By utilizing LLM, the model improves generalization across diverse fault scenarios, addressing the limitations of previous models. The proposed framework incorporates a multimodal data augmentation module, which enhances feature diversity and enriches the representation of complex fault patterns. Furthermore, leveraging large multimodal models facilitates better handling of fault classification by integrating both sequential patterns from time-series data and contextual information from textual descriptions. The textual modality is constructed using templates informed by diagnostic features, allowing the LLM to extract semantically meaningful representations aligned with specific fault characteristics. Experimental results demonstrate the superiority of the proposed multimodal approach, which achieves maximum improvements of 32.647% in ACC and 35.5% in $F1$ -score compared to unimodal methods. In the transferability evaluation, the model achieves a Tr-ACC of 92.295%, demonstrating its robustness and adaptability to unseen datasets. Extensive experiments on industrial-bearing datasets validate the effectiveness of the proposed framework, which outperforms traditional models and highlights its potential for real-world applications.