Flood Detection in Social Media Images using Visual Features and Metadata

Abstract

Images uploaded to social media platforms such as Twitter and Flickr have become a potential source of information about natural disasters. However, due to their lower reliability and noisy nature, it is challenging to automatically identify social media images that genuinely contain evidence of natural disasters. Visual features have been popular for classifying these images while the associated metadata are often ignored or exploited only to a limited extent. To test their potential, we employed them separately to identify social media images with flooding evidence. For visual feature extraction, we utilized three advanced Convolutional Neural Networks (CNNs) pre-trained on two different types of datasets and used a simple neural network for the classification. The results demonstrate that the combination of two types of visual features have a positive impact on distinguishing natural disaster images. From metadata, we considered only the textual metadata. Here, we combined all textual metadata and extracted bi-gram features. Then we employed a Support Vector Machine (SVM) for the classification task. The results show that the combination of the textual metadata can improve the classification accuracy compared to their individual counterparts. The results also demonstrate that although the visual feature approach outperforms metadata approach, metadata have certain capability to classify these images. For instance, the proposed visual feature approach achieved a similar result (MAP = 95.15) compared to the top visual feature approaches presented in MediaEval 2017, the metadata approach outperformed (MAP = 84.52) presented metadata methods. For the experiments, we utilized dataset from MediaEval 2017 Disaster Image Retrieval from Social Media (DIRSM) task and compared the achieved results with the other methods presented (Number of participants = 11) of the task.