A multimodal transfer learning framework for the classification of disaster-related social media images

Rapidly classifying disaster-related social media (SM) images during a catastrophe event is critical for enhancing disaster response efforts. However, the biggest challenge lies in acquiring labeled data for an ongoing (target) disaster to train supervised learning-based models, given that the labeling process is both time-consuming and costly. In this study, we address this challenge by proposing a new multimodal transfer learning framework for the real-time classification of SM images of the target disaster. The proposed framework is based on Contrastive Language-Image Pretraining (CLIP) model, jointly pretrained on a dataset of image-text pairs via contrastive learning. We propose two distinct methods to design our classification framework (1) Zero-Shot CLIP: it learns visual representations from images paired with natural language descriptions of classes. By utilizing the vision and language capabilities of CLIP, we extract meaningful features from unlabeled target disaster images and map them to semantically related textual class descriptions, enabling image classification without training on disaster-specific data. (2) Linear-Probe CLIP: it further enhances the performance and involves training a linear classifier on top of the pretrained CLIP model’s features, specifically tailored to the disaster image classification task. By optimizing the linear-probe classifier, we improve the model’s ability to discriminate between different classes and achieve higher performance without the need for labeled data of the target disaster. Both methods are evaluated on a benchmark X (formerly Twitter) dataset comprising images of seven real-world disaster events. The experimental outcomes showcase the efficacy of the proposed methods, with Linear-Probe CLIP achieving a remarkable 7% improvement in average F1-score relative to the state-of-the-art methods.