A novel lightweight deep attention network for automated nuclei segmentation in histopathology images

Cell nuclei segmentation in histopathological images is essential for developing computer-aided diagnostic (CAD) systems for cancer diagnosis and prognosis. However, this task remains challenging due to poor staining, low contrast, irregular shapes, and overlapping nuclei in histology tissue images. Traditional encoder–decoder-based architectures often struggle to capture fine-grained spatial details while also being computationally expensive. To address these challenges, we propose a novel lightweight deep attention network comprising three residual blocks in the encoder, a bottleneck block, and three wavelet-driven attention blocks in the decoder. The residual blocks used in the encoder effectively extract high-level features, while the bottleneck block captures global multi-resolution features. A newly introduced wavelet-driven attention block in the decoder leverage high-frequency two-dimensional discrete wavelet transform coefficients, that captures the finer edge-level details that are often lost during the encoding process. Evaluated on two publicly available datasets, PanNuke and TNBC, the proposed architecture achieved five-fold cross-validation Jaccard Index scores of $74.47\pm 0.74$ and $71.21\pm 0.82$, respectively, at a 95% confidence level. The proposed architecture has significantly fewer trainable parameters and a smaller model size than existing architectures without compromising its performance. To further validate its efficacy, the model was tested on the MonuSeg dataset as an independent cohort, achieving a Jaccard Index score of $64.17\pm 1.75$. Wilcoxon signed-rank and Scott-Knott ESD tests confirmed that the proposed architecture is statistically superior to existing models. Finally, Grad-CAM heatmaps revealed its superior focus on nuclei regions compared to conventional designs.