AdaptFRCNet: Semi-supervised adaptation of pre-trained model with frequency and region consistency for medical image segmentation

Abstract

Recently, large pre-trained models (LPM) have achieved great success, which provides rich feature representation for downstream tasks. Pre-training and then fine-tuning is an effective way to utilize LPM. However, the application of LPM in the medical domain is hindered by the presence of a large number of parameters and a limited amount of labeled data. In clinical practice, there exists a substantial amount of unlabeled data that remains underutilized. Semi-supervised learning emerges as a promising approach to harnessing these unlabeled data effectively. In this paper, we propose semi-supervised adaptation of pre-trained model with frequency and region consistency (AdaptFRCNet) for medical image segmentation. Specifically, the pre-trained model is frozen and the proposed lightweight attention-based adapters (Att_Adapter) are inserted into the frozen backbone for parameter-efficient fine-tuning (PEFT). We propose two consistency regularization strategies for semi-supervised learning: frequency domain consistency (FDC) and multi-granularity region similarity consistency (MRSC). FDC aids in learning features within the frequency domain, and MRSC aims to achieve multiple region-level feature consistencies, capturing local context information effectively. By leveraging the proposed Att_Adapter along with FDC and MRSC, we can effectively and efficiently harness the powerful feature representation capability of the LPM. We conduct extensive experiments on three medical image segmentation datasets, demonstrating significant performance improvements over other state-of-the-art methods. The code is available at https://github.com/NKUhealong/AdaptFRCNet.