Deep learning in surgical process modeling: A systematic review of workflow recognition

Abstract

Objective: The application of artificial intelligence (AI) in health care has led to a surge of interest in surgical process modeling (SPM). The objective of this study is to investigate the role of deep learning in recognizing surgical workflows and extracting reliable patterns from datasets used in minimally invasive surgery, thereby advancing the development of context-aware intelligent systems in endoscopic surgeries. Methods: We conducted a comprehensive search of articles related to SPM from 2018 to April 2024 in the PubMed, Web of Science, Google Scholar, and IEEE Xplore databases. We chose surgical videos with annotations to describe the article on surgical process modeling and focused on examining the specific methods and research results of each study. Results: The search initially yielded 2937 articles. After filtering on the basis of the relevance of titles, abstracts, and content, 59 articles were selected for full-text review. These studies highlight the widespread adoption of neural networks, and transformers for surgical workflow analysis (SWA). They focus on minimally invasive surgeries performed with laparoscopes and microscopes. However, the process of surgical annotation lacks detailed description, and there are significant differences in the annotation process for different surgical procedures. Conclusion: Time and spatial sequences are key factors determining the identification of surgical phase. RNN, TCN, and transformer networks are commonly used to extract long-distance temporal relationships. Multimodal data input is beneficial, as it combines information from surgical instruments. However, publicly available datasets often lack clinical knowledge, and establishing large annotated datasets for surgery remains a challenge. To reduce annotation costs, methods such as semi supervised learning, self-supervised learning, contrastive learning, transfer learning, and active learning are commonly used.