Open-world object detection with multi-dataset image–label matching

Abstract

In real-world scenarios, many categories appear in target scenes that were not encountered during training, making existing video object detection methods unsuitable for open-world applications. This paper proposes an open-world object detection method based on multi-dataset image–label matching to tackle the challenges of open-world object detection. First, a multi-dataset image–label matching training strategy is proposed, which aligns image features with label text features from multiple datasets, an innovative matching classification loss function is designed to guide model training. Then, an image–label deep fusion module is constructed to strengthen the model’s ability to understand the correspondence between visual and textual descriptions, thereby improving the accuracy of matching label texts to corresponding regions in images. A decoupled, staged training method is employed, independently training the proposal generation and category classification stages to better adapt to the diversity and uncertainty of open-world scenarios. Finally, extensive comparative and ablation experiments validate the proposed method’s effectiveness on the open-world dataset LVIS, achieving an average improvement of about 2 percentage points over baseline methods in various evaluation metrics. Additionally, visualizations across different scenes are presented to intuitively demonstrate the method’s efficacy and advanced performance.