GAN-based alignment multilevel attention network for multimodal classification of crisis events in social media

Social media has changed the way information is shared and disseminated, especially in the context of disaster events detection. A data-intensive vision-language model has achieved state-of-the-art results on emergency response detection tasks. Until now, most event detection methods in this area have focused on joint analysis approaches for text and image but have overlooked the disparity between text and image information on social media. In this paper, we propose an approach employing a pre-trained model to construct multilevel inputs with multilevel features for each modality. This approach extracts subtle connections between different types of social media content in a hierarchical manner, utilizing a novel cross-attention mechanism for multimodal representation fusion. Additionally, our method incorporates a GAN-based cross-modal alignment module, further enhancing the model's capability in handling multimodal data. Our method outperforms a strong baseline on two crisis-related tasks and shows great potential to improve disaster response and management efficiency and accuracy, providing strong support for future disaster relief efforts.