AMS: Attention Map Seeds for enhancing interactive segmentation

Abstract

Interactive segmentation has advanced significantly. A key challenge in these methods is the selection of query seeds, which often rely on previous seeds’ definitions. This can make it difficult to identify segmentation results that deviate from the ground truth, as users focus on regions with clear positive or negative samples. To address this, we propose a new approach using a visual heatmap-based interaction mechanism with Attention Map Seeds (AMS). AMS is generated by computing the difference between the Segment Gradient-weighted Class Activation Mapping (Seg-Grad-CAM) heatmap and the ground truth. Users receive sparse query seeds along with visual explanations in each interaction round, allowing them to observe and distinguish subtle differences between segmentation results and the ground truth across the entire image. We tested our algorithm on four publicly available datasets. Results show that, on average, AMS achieved a 4.1% higher $Dice$ score per experimental round compared to the state-of-the-art (SOTA) in single-query seed testing. In multi-query seed testing, AMS needed 1.28 fewer clicks on average to achieve convergence precision compared to SOTA.