Style transformation and distance map guided nucleus instance segmentation via multi task learning

Abstract

Instance segmentation of nuclei in histopathological images is hindered by three critical challenges: overlapping nuclei, domain shift caused by staining variability, and generalization across diverse multi-organ datasets. To address these issues, we propose a unified multi-task learning framework for nucleus instance segmentation that integrates style transformation and distance map-guided segmentation. Our architecture employs multi-dilated residual blocks and encoder–decoder attention gates to capture multi-scale features and preserve fine nuclear details, while a transformer in the bottleneck enhances contextual understanding and models long-range dependencies. The network incorporates dual heads for semantic segmentation and distance-map prediction, effectively addressing overlapping nuclei. Additionally, a histogram-based, reference-guided stain normalization module mitigates domain shift caused by staining variability, and when combined with our robust model architecture, it enhances the overall generalization ability across multi-organ datasets. Experimental results demonstrate our method’s superior performance over existing segmentation approaches. The source code is available at https://github.com/eyob12/MTL-NucleusSeg.