Prompt-guided and degradation prior supervised transformer for adverse weather image restoration

Abstract

The restoration of images affected by adverse weather conditions is hindered by two main challenges. The first is the restoration of fine details in severely degraded regions. The second is the interference between different types of degradation data during the model training process, which consequently reduces the restoration performance of the model on individual tasks. In this work, we propose a Transformer-based All-in-one image restoration model, called PDFormer, to alleviate the aforementioned issues. Initially, we designed an effective transformer network to capture the global contextual information in the image and utilize this information to restore the locally severely degraded regions better. Additionally, to alleviate the interference between different types of degraded data, we introduced two specialized modules: the Prompt-Guided Feature Refinement Module (RGRM) and the Degradation Mask Supervised Attention Module (MSAM). The former employs a set of learnable prompt parameters to generate prompt information, which interacts with the degraded feature through cross-attention, enhancing the discriminative ability of different degraded features in the latent space. The latter, under the supervision of the degraded mask prior, assists the model in differentiating between different degradation types and locating the regions and sizes of the degradations. The designs above permit greater flexibility in handling specific degradation scenarios, enabling the adaptive removal of different degradation artifacts to restore fine details in images. Performance evaluation on both synthetic and real data has demonstrated that our method surpasses existing approaches, achieving state-of-the-art (SOTA) performance.