SAM-Assisted Temporal-Location Enhanced Transformer Segmentation for Object Tracking with Online Motion Inference

Abstract

Current transformer-based trackers typically represent targets using bounding boxes. However, bounding boxes do not accurately describe the target and uncontrollably contain most background pixels. This paper proposes a Segment Anything Model (SAM)-Assisted Temporal-Location Enhanced Transformer Segmentation for Object Tracking with Online Motion Inference. First, a novel transformer-based temporal-location enhanced segmentation method is proposed. The target temporal features are clustered into foreground–background tokens utilizing a mask to capture discriminative information distribution. Then, the suitable positional prompts are learned in the proposed mask prediction head to establish the mapping between target features and localization, which enhances the specific foreground weights for precise mask generation. Second, a temporal-based motion inference module is proposed. It fully utilizes the target temporal state to construct an online displacement model inferring motion relationships of the target between frames and providing robust position prompts for the segmentation process. We also introduce SAM for initial mask generation. Precise pixel-level object tracking is achieved by combining segmentation and localization within a unified process. Experimental results demonstrate that the proposed method yields competitive performance compared to existing approaches.