Multimodal artificial intelligence approaches using large language models for expert-level landslide image analysis

Abstract

Climate change exacerbates natural disasters, demanding rapid damage and risk assessment. However, expert-reliant analyses delay responses despite drone-aided data collection. This study develops and compares multimodal AI approaches using advanced large language models (LLMs) for expert-level landslide image analysis. We tackle landslide-specific challenges: capturing nuanced geotechnical reasoning beyond data digitization (specific to geological features and risk assessment), developing specialized transfer learning and data augmentation to mitigate data scarcity and geological diversity in landslide imagery, and establishing tailored evaluation metrics including geological accuracy, risk validity, and decision utility for landslide analysis. Evaluating a visual question answering-large language model (VQA-LLM) hybrid (sequential visual processing) and a multimodal large language model (MLLM, simultaneous vision/text processing) shows that MLLM excels in disaster identification, while the VQA-LLM hybrid demonstrates superior performance in risk assessment, thereby informing optimal AI design choices. Our methodology, structuring 30+ years of expert commentary for AI training and employing a comprehensive evaluation framework including standard text metrics, LLM-based semantic analysis, and expert domain assessment, highlights the potential of hybrid systems and addresses knowledge transfer in data-sparse domains.