MarIns3D: An open-vocabulary 3D instance segmentation model with mask refinement

Abstract

Open-vocabulary 3D instance segmentation has gained significant attention due to its potential role in scene perception. Existing methods typically involve two stages: generating class-agnostic 3D instance masks using segmentation models, followed by semantic classification of these masks. However, poor classification performance often stems from low-quality masks in the first stage. This paper proposes two key components to optimize the mask generation process: a dynamic offset module and a projection consistency loss. By dynamically adjusting sampling point positions, query points can capture key scene features to generate high-quality masks. Then the projection consistency loss compares these 3D instance masks with ground truth in 2D projections to refine them, improving segmentation accuracy. Experimental results on the ScanNetV2 validation set show that MarIns3D outperforms SOLE on zero-shot segmentation, with a 1.8 % and 1.7 % improvement in AP25 and AP50, respectively, and also demonstrates enhanced open-set segmentation capabilities. These results confirm our model’s superior mask quality and segmentation performance. Furthermore, ablation studies verify that the synergy between the dynamic offset module and the projection consistency loss is crucial for these enhancements.