Zero-shot object detection based on cross-modal guided clustering

At present, contrastive learning has been widely used in Zero-Shot Object Detection (ZSD) and proved to be able to reduce inter-class confusion. However, existing ZSD clustering algorithms operate spontaneously, without effective guidance, and may therefore cluster in the wrong places, as they are constrained to a single visual modality. It is difficult to achieve cross-modal alignment, and textual guidance can help achieve ideal visual clustering. In view of the above problems, this paper proposes a novel zero-shot object detection method based on cross-modal guided clustering, which is a new method for ZSD that combines image-to-image contrast with an auxiliary image-to-text contrast during training. Firstly, an instance-level cross-modal contrastive embedding (ICCE) loss is proposed, by which text similarities are used as dynamic weights to guide the modal to focus on the most confusing categories, and ignoring low similarity ones. A cross-level cross-modal contrastive embedding (CCCE) loss based on ICCE is also designed to provide an ideal guided cluster center. Finally, a cross-modal triplet loss (CTL) is introduced to divide anchors into positive and negative anchors to address the problem that negative samples are difficult to cluster effectively. The first two highlight class-level similarities to avoid misclassification in the most confusing categories, while the last focuses on capturing the most challenging cases to ensure it can handle difficult instances effectively. Experimental tests and comparisons are conducted with the current advanced methods on three baseline databases, and the results demonstrate that the proposed method can achieve a better detection effect, especially when the number of training categories is limited.