SLIP-flood: Soft-combination of Swin Transformer and Lightweight Language-Image Pre-training for Flood Images Classification

Abstract

Flood monitoring is a complex task involving multimodal data mining and multitask collaboration. In order to leverage the role of multimodal data in flood management, conducting visual-language pretraining (VLP) in the field of flood disaster monitoring and obtaining foundational pretraining models that are suitable for multiple downstream flood-related tasks is an urgent problem that needs to be addressed. This paper introduces SLIP-Flood, an innovative VLP framework supporting flood image classification, image-text retrieval, and auxiliary text classification. To overcome the limitations of existing cross-modal models that rely on small datasets and lack robustness, we have constructed two specialized datasets for the first time: 1) FloodMulS for the Flood Image Classification Model (FICM), and 2) FloodIT for the Flood Text-Image Retrieval Model (FTIRM). Traditional models employ “Hard Categorization Strategy (HC)” for image classification, neglecting the impacts of “Categorization Ambiguity.” To improve performance, we propose a “Soft Categorization Strategy.” Furthermore, traditional models focus on unimodal (image) information, not fully utilizing joint image-text information. We address this with a “Soft Combination” to integrate FICM and FTIRM, termed SCSC. Experimental results show SCSC improves SLIP-Flood’s performance: a 7.62% increase in the F1 score on FICM compared to HC, and a 0.35% increase in FTIRM’s F1 score based on FICM. SLIP-Flood also achieves a maximum recall of 89.24% in image-text retrieval and shows promise in auxiliary flood text classification. Relevant resources are available at https://github.com/muhan-yy/SLIP-Flood.git.