Achieving fair medical image segmentation in foundation models with adversarial visual prompt tuning

Abstract

Recent advances in deep learning have significantly enhanced medical image analysis capabilities. Medical image segmentation, a critical application in this domain, enables precise delineation of anatomical structures and pathological regions, substantially supporting clinical decision-making. However, current segmentation methods primarily optimize for overall performance without considering disparities across demographic groups, raising important fairness concerns. To address this gap, we propose Adversarial Visual Prompt Tuning (AdvVPT), a parameter-efficient approach that enhances fairness in foundation models for medical image segmentation. AdvVPT introduces trainable visual prompts within the image encoder while keeping the backbone frozen, requiring only 0.812M additional parameters. These prompts are optimized through adversarial training to absorb demographic-specific biased information from image embeddings, achieved by maximizing prediction errors for sensitive attributes and increasing embedding distances between visual prompts and image features. Experimental evaluation on the Harvard-FairSeg dataset demonstrates that AdvVPT achieves state-of-the-art fairness performance across multiple demographic attributes. For racial fairness, AdvVPT achieves an ES-Dice score of 0.8996 and an ES-IoU score of 0.8222 on optic cup segmentation, substantially outperforming existing methods. For gender fairness using the SAT backbone, AdvVPT achieves an ES-Dice of 0.9297 and ES-IoU of 0.8614, demonstrating both superior performance and improved balance between male and female subgroups.