A spatiotemporal dynamic fusion network for surgical action recognition

Surgical action recognition is crucial and tough in intelligent surgical robots, enabling these systems to accurately identify and interpret the ongoing actions within a surgical procedure. By recognizing the action state in real-time, the robot can provide immediate feedback and make adjustments to ensure the precision and safety of the surgery. However, it faces several challenges, such as the temporal complexity of surgical actions, the fine operation steps and subtle changes in surgical movements. Thus, a dynamic attention mechanism has been proposed to capture the temporal correlation between the current frame and the previous frames from video sequence. Furthermore, a spatiotemporal dynamic fusion network comprising two specialized modules has been proposed. The first module, Double Bi-level Routing Attention (DBRA), is designed to extract the most pertinent spatial and temporal features. While the other module, CNN-LSTM, is dedicated to delivering comprehensive spatiotemporal information. Experiments have been conducted on both a neurosurgical dataset Neuro67 and a public dataset Suturing to demonstrate the performance of the proposed method. The results indicate that the proposed method has achieved superior performance on the hard issue, achieving AP on Neuro67 and ACC on Suturing of 76.9% and 85.5%, leading by 6.7% and 1.2% respectively, with the modules effectively focusing on dependency relationships both within regions of a frame and across frames in video sequence.