Multi-task learning for accurate and efficient nucleus instance segmentation based on ordinal regression

Abstract

Nucleus instance segmentation is a critical prerequisite in many microscopy-related research fields, including pathology, drug discovery and functional genomics. The biological tasks involved depend on highly accurate and readily available nucleus segmentation results. However, both manual and existing computer-assisted methods face challenges in balancing accuracy and efficiency due to the diverse sizes, shapes and morphologies of nuclei. Additionally, some nuclei are often clustered and overlapping, which imposes higher demands on segmentation methods. Here, we present an ordinal regression-based nucleus instance segmentation method with multi-task learning that leverages rich instance-aware information encoded within spatial-based ordinal rankings. These ordinal rankings are generated and predicted by our proposed Distance Grading Decrease (DGD) strategy and EfficientNet-based lightweight network, W-Net, respectively. Combined with pixel-level foreground probabilities, these rankings are utilized to separate clustered nuclei and achieve accurate segmentation through a marker-controlled watershed algorithm. Our method demonstrates state-of-the-art accuracy and efficiency compared to others, as validated on two independent multi-tissue histology image datasets.