Align knowledge with time-series: Cross-modal domain knowledge activation for LLM-enabled zero-shot fault diagnosis

The existing zero-shot fault diagnosis methods identify unseen categories that have no training samples by predicting fault attributes from samples. Although these methods alleviated the data scarcity issue, they raised a new challenge, i.e., professional fault attributes annotation. We recognize that the essence of fault attributes lies in describing the connections and differences between categories. Therefore, we propose a novel LLM-enabled zero-shot fault diagnosis paradigm, and the large language models (LLMs) fine-tuned based on domain-specific knowledge can capture similar information to replace manual annotation. It unlocks the potential of LLMs to handle zero-shot tasks related to industrial time-series data in a cross-modal manner. It aims to address the burden of semantic knowledge annotation posed by the existing attribute-enabled paradigm. Moreover, the domain shift problem (DSP) arising from the shortage of training samples for unseen faults is also tackled by leveraging the cross-modal activation of relevant knowledge that has been learned from domain-specific documents. Firstly, to address the issue that LLMs pretrained on general knowledge are lacking in the knowledge of the industrial field, we design prompts for industrial faults and fine-tune the LLM with domain knowledge from diagnosis reports. Subsequently, considering that LLMs lack the ability to process time-series data, we design a cross-modal transformation module to align the time-series modality with the text modality. Moreover, we propose a knowledge distillation strategy to further align these two modalities, so the unseen fault text descriptions can serve as substitutes for the unavailable samples to address the DSP. We conduct experiments on a real thermal power plant, and the proposed method achieves an average improvement of 9.83% in terms of the diagnosis accuracy of unseen faults.