BiasPruner: Mitigating bias transfer in continual learning for fair medical image analysis

Abstract

Continual Learning (CL) enables neural networks to learn new tasks while retaining previous knowledge. However, most CL methods fail to address bias transfer, where spurious correlations propagate to future tasks or influence past knowledge. This bidirectional bias transfer negatively impacts model performance and fairness, especially in medical imaging, where it can lead to misdiagnoses and unequal treatment. In this work, we show that conventional CL methods amplify these biases, posing risks for diverse patient cohorts. To address this, we propose BiasPruner, a framework that mitigates bias propagation through debiased subnetworks, while preserving sequential learning and avoiding catastrophic forgetting. BiasPruner computes a bias attribution score to identify and prune network units responsible for spurious correlations, creating task-specific subnetworks that learn unbiased representations. As new tasks are learned, the framework integrates non-biased units from previous subnetworks to preserve transferable knowledge and prevent bias transfer. During inference, a task-agnostic gating mechanism selects the optimal subnetwork for robust predictions. We evaluate BiasPruner on medical imaging benchmarks, including skin lesion and chest X-ray classification tasks, where biased data (e.g., spurious skin tone correlations) can exacerbate disparities. Our experiments show that BiasPruner outperforms state-of-the-art CL methods in both accuracy and fairness. Code is available at: BiasPruner.