A Hierarchical ViT With Dynamic Window Shift Unit and Curriculum Learning for Remote Sensing Image Scene Classification

Abstract

The remote sensing image (RSI) scene classification is currently a popular research topic among many remote sensing tasks. However, RSI scene classification still faces challenges such as complex multiscale key features concentrated in different local regions, large interclass imbalance, and intraclass variation in insufficient well-labeled RSI samples. To address these challenges, we proposed a novel RSI scene classification method based on an improved vision transformer. This method has better multiscale feature representation ability due to the improved hierarchical vision transformer structure, in which a feature map fusion layer produces feature maps of different sizes, and a window transformer block with dynamic window shift unit actively shifts to the local region with dense information, flexibly extracting and associating key features with multiscale in different input regions. Furthermore, we design a curriculum transfer learning framework to alleviate the problems of lacking well-labeled training samples, intraclass variation, and interclass imbalance during the training process of the improved vision transformer. This framework employs a dual-criteria difficulty evaluator to evaluate training samples and provides supplementary supervision to the model by generating training task schedules. Finally, experimental results demonstrate that the proposed vision transformer model achieves rapid convergence and superior performance in RSI scene classification task.