Hierarchical Fusion Transformer for Multimodal Ground-Based Cloud Type Classification

Abstract

Existing methods for multimodal ground-based cloud type classification are dominated by convolutional neural networks, and it fails to capture long-range dependencies. In this article, we propose a novel Transformer-based architecture named hierarchical fusion transformer (HFT) for multimodal ground-based cloud type classification, which leverages the advantages of self-attention and cross-attention to learn long-range dependencies and effectively fuse cloud images and meteorological element information. Specifically, we propose visual and meteorological joint-transformer (VM Joint-Trans) to capture global context across modalities and present visual and meteorological cross-transformer (VM Cross-Trans) to align different modalities and reduce their inconsistencies. We design a hierarchical architecture to perform comprehensive fusion using VM Joint-Trans and VM Cross-Trans. Meanwhile, we propose the novel multimodal contrastive learning, which not only constrains the tokens of cloud images and meteorological element information in the same layer, but also the tokens from the same modality in different layers, thereby improving the discriminative ability of model and reducing the modality gap. Furthermore, we release the large-scale multimodal ground-based cloud database, containing 10 000 multimodal samples with seven categories. To the best of the authors’ knowledge, it is the largest database for multimodal ground-based cloud type classification. Experimental results validate the effectiveness of the proposed HFT for multimodal ground-based cloud type classification.