Deep learning-based semantic segmentation of abdominal wall layers during direct optical trocar insertion: toward AI-assisted surgical access.

Abstract

Background: The initial entry phase of minimally invasive surgery-direct optical trocar insertion (DOTI)-remains high risk due to potential spatial disorientation. To our knowledge, this pilot study investigates a deep learning-based semantic segmentation framework for identifying critical abdominal wall layers during DOTI.

Methods: From 10 DOTI procedures, 951 frames were extracted and pixel-wise annotated into three anatomical classes-rectus muscle, rectus sheath, and peritoneum-plus background. A DeepLabV3+ architecture with ResNet-50 backbone was trained using five-fold cross-validation. Performance was assessed by mean Intersection over Union (mIoU), mean Dice coefficient (mDice), precision, recall, and F1-score.

Results: The model achieved a mean IoU of 0.794 (95% CI 0.785-0.802) and a mean Dice of 0.883 (95% CI 0.878-0.889). Class-specific Dice scores were 0.923 for rectus muscle, 0.903 for rectus sheath, and 0.824 for peritoneum. Confusion matrix analysis showed minimal direct confusion between anatomical tissue classes (0.0-0.3%), with most errors confined to tissue-background boundaries.

Conclusion: This pilot single-center study demonstrates the technical feasibility of deep learning-based semantic segmentation for extra-abdominal tissue-layer identification during optical trocar insertion. Although the model showed consistent performance across five-fold cross-validation, the small patient cohort limits generalizability, and larger multi-center and prospective studies are required before clinical translation. The proposed framework provides a preliminary basis for future investigation of AI-assisted surgical guidance.