A multimodal expert system for the intelligent monitoring and maintenance of transformers enhanced by multimodal language large model fine-tuning and digital twins

Abstract

The development of multimodal large models and digital twin technology is set to revolutionise the methods of intelligent monitoring and maintenance for transformers. To address the issues of low intelligence level, single application mode, and poor human–machine collaboration in traditional transformer monitoring and maintenance methods, an intelligent monitoring and maintenance digital twin multimodal expert reasoning system, fine-tuned on visual language-based large models, is proposed. The paper explores the modes and methods for implementing intelligent monitoring and maintenance of transformers based on multimodal data, large models, and digital twin technology. A multimodal language large model (MLLM) framework for intelligent transformer maintenance, grounded on the Large Language and Vision Assistant model, has been designed. To enable large models to understand and reason about image annotation areas, an adaptive grid-based positional information processor has been designed. To facilitate the compatibility and learning of large models with transformer Dissolved Gas Analysis data, a heterogeneous modality converter based on the Gram–Schmidt angular field has been developed. For the unified modelling and management of multimodal reasoning and comprehensive resource integration in human–machine dialogue, a central linker based on an identity resolution asset management shell has been designed. Subsequently, a visual-language multimodal dataset for transformer monitoring and maintenance was constructed. Finally, by fine-tuning parameters, a multimodal expert reasoning system for intelligent transformer monitoring and maintenance was developed. This system not only achieves real-time monitoring of the transformer's operational status but also generates maintenance strategies intelligently based on operational conditions. The expert system possesses robust human–machine dialogue capabilities and reasoning generation abilities. This research provides a reference for the deep integration of MLLM and digital twin in industrial scenarios, particularly in the application modes of intelligent operation and maintenance for transformers.