Improving vision-language models through intra-modal contrastive learning-based hard sample mining

Driving environmental perception is a core component of autonomous driving systems. Recently, emerging vision-language detectors, known for their superior detection accuracy, have gradually replaced traditional detectors and have been increasingly applied in open-world driving scenarios. However, these detectors still face challenges regarding the missed detection of hard positive samples. This study identifies that a primary cause of this problem is the rejection of hard samples due to their low cross-modal consistency. To address this challenge, this work proposes a contrastive learning strategy based on a hard sample prototype memory bank to recall the potential positive samples. Additionally, to enhance the representational capacity of the detection network, an instance-level contrastive learning loss is introduced. This loss aligns the feature representations of the same instance across the deep and shallow network layers, thereby improving the ability of shallow layers to extract features from hard samples. Experimental results demonstrate that the proposed method achieves outstanding detection accuracy and is highly effective in complex urban road scenarios. The code and trained models are available at https://github.com/unbelieboomboom/HSMG_DINO.