Quantitative Detection of Entotic Cell-In-Cell Structures Using Deformable Segmentation and Deep Learning

Abstract

Cell-in-cell (CIC) structures are among the most intriguing cellular phenomena occasionally observed in human cancer specimens. Once regarded as incidental findings, accumulating evidence has linked specific CIC subtypes, particularly entosis, to tumor progression and patient prognosis. Despite growing interest, systematic investigation of entosis remains limited due to the labor-intensive nature of manual identification and analysis. Computational approaches are, therefore, needed to enable scalable and reproducible entosis detection. In this study, we developed and evaluated five morphology-driven deformable segmentation models alongside a YOLOv8 deep learning–based detection framework for automated entotic cell identification in BxPC3 (pancreatic) and MCF7 (breast) cancer cell lines. The deformable models were designed to capture complementary morphological characteristics, including entropy, spatial proximity, contour topology, and circularity. Comparative evaluation showed that deformable Models A and C achieved the highest sensitivity, with recall values ranging from 0.91 to 0.94 and F1-scores between 0.81 and 0.83, demonstrating robust performance across heterogeneous entotic morphologies. YOLOv8 achieved high overall accuracy (0.97) and specificity (0.98), indicating strong background discrimination, but exhibited lower recall (0.65) and F1-score (0.59), reflecting a conservative detection profile under extreme class imbalance, where entotic events comprised approximately 1% of all observed cells. While deformable models provided higher sensitivity and detailed morphological segmentation, YOLOv8 offered advantages in computational efficiency and rapid inference. Together, these findings highlight the complementary strengths of morphology-driven segmentation and deep learning–based detection, and support the future development of scalable hybrid frameworks for automated entosis analysis.