SAM-HovNet: SAM-Enhanced Multi-Class Nucleus Segmentation and Classification in Hepatocellular Carcinoma Slides.

Abstract

Hepatocellular carcinoma (HCC) is the most common primary liver cancer and a leading cause of cancer-related mortality. Manual nuclear analysis in HCC histopathology is labor-intensive and prone to interobserver variability, underscoring the need for robust automated solutions. Although deep learning methods for nuclei segmentation and classification have progressed substantially, their performance on liver tissue remains suboptimal. To address this limitation, we propose SAM-HovNet, a unified framework that enhances HoverNet by integrating the SAM foundation model, Squeeze-and-Excitation modules, and multi-scale feature representations for multi-class nucleus segmentation and classification in HCC slides. Experiments on the HCCNuc dataset demonstrate that SAMHovNet achieves an image-level mean F1-score of 0.577 and an image-level mean panoptic quality (PQ) of 0.521. Furthermore, SAM-HovNet demonstrates strong generalizability, significantly outperforming competing approaches on the public PanNuke dataset. These results highlight the robustness of SAM-HovNet and its potential to support downstream nuclear characterization in HCC histopathology.