CellKAN: Cellular multi-attention Kolmogorov-Arnold networks for nuclei segmentation in histopathology images

Abstract

This paper presents CellKAN, a novel medical image segmentation network for nuclei detection in histopathological images. The model integrates a Multi-Scale Conv Block (MSCB), Hybrid Multi-Dimensional Attention (HMDA) mechanism, and Kolmogorov-Arnold Network Block (KAN-Block) to address challenges like missed tiny lesions, heterogeneous morphology parsing, and low-contrast boundary inaccuracies. MSCB enhances multi-scale feature extraction via hierarchical refinement, while HMDA captures cross-channel-spatial dependencies through 3D convolution and dual-path pooling. KAN-Block replaces linear weights with learnable nonlinear functions, enhancing model interpretability and reducing the number of parameters. Evaluated on MoNuSeg, PanNuke, and an In-house gastrointestinal dataset, CellKAN achieves Dice coefficients of 82.91 %, 83.50 %, and 71.38 %, outperforming state-of-the-art models (e.g., U-KAN, nnUNet) by 1.29–4.49 %. Ablation studies verify that MSCB and HMDA contribute 0.35 % and 0.48 % Dice improvements on PanNuke, respectively. The model also reduces parameters compared to nnUNet while maintaining high accuracy, balancing precision and efficiency. Visual results demonstrate its superiority in noise suppression, boundary delineation, and structural integrity, highlighting its potential for clinical pathological analysis.