Large multimodal model assisted underground tunnel damage inspection and human-machine interaction

Abstract

Artificial Intelligence is playing an increasingly important role in tunnel inspection as a core driver of the new generation of engineering. Traditional methods are difficult to directly generate human linguistic information and lack valid messages extracted from different modalities. This paper proposes Damage LMM, a multimodal damage detection model that can handle images or videos as well as text inputs, to realize fast damage identification and human-computer interaction. The visual instruction database is first created from real damage data collected using different visual sensors and captions extracted by a regional convolutional neural network. The basic language model is then fine-tuned into a specialised Damage LMM, which enhances user instructions by integrating virtual prompt injection and system messages. Finally, the enhanced prompts are processed through the tuned multimodal model to generate a detailed visual description of the damage. The performance of the method is evaluated using a real tunnel dataset, and the results show that it has better robustness and accuracy than other models in multimodal data, with an accuracy of 0.93 for the in-domain image data and a contextual correlation of 0.94. The proposed method can effectively identify tunnel defects and realize multimodal user interaction functions with a moderate number of markers and a short delay time, which will greatly help engineers to quickly obtain effective information and assess the degree of damage at the tunnel inspection site.