Automatic segmentation of prostate and organs at risk in CT images using an encoder–decoder structure based on residual neural network

Abstract

Accurate segmentation of the prostate and surrounding organs at risk (OARs) from CT scans is critical for radiotherapy treatment planning in prostate cancer. However, manual segmentation is time-consuming and prone to variability. This paper proposes a deep learning-based approach using a pre-trained ResNet-18 combined with an encoder–decoder structure based on DeepLabv3+. The method automates the segmentation of the prostate, bladder, and rectum in male pelvic CT scans, achieving precise and efficient results without requiring preprocessing or extensive manual refinement. Evaluated on 100 CT scans using 10-fold cross-validation, the model demonstrates strong performance (Dice Similarity Coefficient (DSC) and Hausdorff Distance (HD)) on prostate (DSC: $84.32\pm 4.88\text{\%}$, HD: $3.95\pm 0.60\mathrm{mm}$), bladder (DSC: $86.53\pm 3.66\text{\%}$, HD: $4.58\pm 0.72\mathrm{mm}$), and rectum (DSC: $83.92\pm 4.18\text{\%}$, HD: $2.99\pm 0.40\mathrm{mm}$) segmentation. Additionally, a user-friendly MATLAB application is developed to automate the segmentation process. This approach has the potential to improve treatment planning efficiency, accuracy, and consistency for better patient outcomes.