SimPS-Net: Simultaneous Pose & Segmentation Network of Surgical Tools

Image-based detection and localisation of surgical tools has received significant attention due to the development of rele- vant deep learning techniques, along with recent upgrades in computational capabilities. Although not as accurate as optical trackers [1], image-based methods are easy to deploy, and require no surgical tool redesign to accommodate trackable markers, which could be beneficial when it comes to cheaper, “off-the-shelf” tools, such as scalpels and scissors. In the operating room however, these techniques suffer from drawbacks due to the presence of highly reflective or featureless materials, but also occlusions, such as smoke and blood. Furthermore, networks often utilise tool 3D models (e.g. CAD data), not only for the purpose of point correspon- dence, but also for pose regression. The aforementioned “off- the-shelf” tools are scarcely accompanied by such prior 3D structure data. Ultimately, in addition to the above hindrances, estimating 3D pose using a monocular camera setup, poses a challenge in itself due to the lack of depth information. Con- sidering these limitations, we present SimPS-Net, a network capable of both detection and 3D pose estimation of standard surgical tools using a single RGB camera.