Abnormal Scene Classification using Image Captioning Technique: A Landslide Case Study

Abstract

Disaster occurs in various regions, which affects human life and activities. For instance, the one disaster we faced in the mountainous area was a landslide that could occur for various reasons, such as heavy precipitation, failure of slope stability, or earthquake. After this event, humans must evaluate the damage and the damaged area to recover it. Therefore, we must focus on autonomous damage detection to evaluate the damaged region and plan for the decision-maker for recovery. This research discusses a new technique for detecting dam-age in disaster situations, specifically landslides. The technique combines a transformer model for object detection and a vision encoder-decoder for image captioning. By using language to detect abnormal scenes rather than just visual features, the proposed method can accurately predict the extent of damage in the affected region. The results of this study show that the proposed method outperforms the traditional ResNet50 classification method in terms of accuracy, AUC, precision, recall, and F1 score. These findings suggest that the proposed technique could be an effective tool for evaluating the scope of damage after a disaster, which could help the recovery process.