STSLAM: Robust visual SLAM in dynamic scenes via image segmentation and instance tracking

Although visual simultaneous localization and mapping (SLAM) has made significant progress in localization accuracy, its robustness can be further improved. The primary reason for this is the insufficient modeling of dynamic instances, which leads to tracking failures for current SLAM methods in dynamic scenes. Furthermore, the lack of semantic information is also a problem in the traditional visual SLAM field. To solve these problems, this paper proposes a visual SLAM algorithm called segmentation and tracking SLAM (STSLAM). We apply image segmentation and instance tracking to visual SLAM. The image segmentation and instance tracking task is achieved through a video panoptic segmentation algorithm. By integrating the learning-based algorithm into the SLAM system, STSLAM not only achieves motion estimation for each dynamic instance but also introduces novel factors for factor graph construction to constrain these dynamic instances. Meanwhile, we use the learning-based algorithm to assign semantics to the map and build a panoptic point cloud map. Finally, ablation studies and comparative experiments are conducted on the KITTI, TUM RGB-D and Bonn RGB-D Dynamic dataset, which verify the effectiveness of the STSLAM method and achieve state-of-the-art performance.