Multilayer Encoder-Decoder Network for 3D Nuclear Segmentation in Spheroid Models of Human Mammary Epithelial Cell Lines

Nuclear segmentation is an important step in quantitative profiling of colony organization in 3D cell culture models. However, complexities arise from technical variations and biological heterogeneities. We proposed a new 3D segmentation model based on convolutional neural networks for 3D nuclear segmentation, which overcomes the complexities associated with non-uniform staining, aberrations in cellular morphologies, and cells being in different states. The uniqueness of the method originates from (i) volumetric operations to capture all the threedimensional features, and (ii) the encoder-decoder architecture, which enables segmentation of the spheroid models in one forward pass. The method is validated with four human mammary epithelial cell (HMEC) lines—each with unique genetic makeup. The performance of the proposed method is compared with the previous methods and is shown that the deep learning model has a superior pixel-based segmentation, and an F1-score of 0.95 is reported.