Knowledge-Driven Framework for Anatomical Landmark Annotation in Laparoscopic Surgery

Accurate and reliable annotation of anatomical landmarks in laparoscopic surgery remains a challenge due to varying degrees of landmark visibility and changing shapes of human tissues during a surgical procedure in videos. In this paper, we propose a knowledge-driven framework that integrates prior surgical expertise with visual data to address this problem. Inspired by visual reasoning knowledge of tool-anatomy interactions, our framework models a spatio-temporal graph to represent the static topology of tool and tissue and dynamic transitions of landmarks’ temporal behavior. By assigning explainable features of the surgical scene as node attributes in the graph, the surgical context is incorporated into the knowledge space. An attention-guided message passing mechanism across the graph dynamically adjusts the focus in different scenarios, enabling robust tracking of landmark states throughout the surgical process. Evaluations on the clinical dataset demonstrate the framework’s ability to effectively use the inductive bias of explainable features to label landmarks, showing its potential in tackling intricate surgical tasks with improved stability and reliability.