CLIP-AFIR: A Contrastive Language-Image Pretraining Model for Accurate Fish Individual Fine-grained Re-identification

Abstract

Similar to identifying different human individuals, accurately determining ”who is who” is a critical component in intelligent aquaculture, with broad applications in disease analysis, growth monitoring, and other aspects. The accuracy of fish Re-Identification (ReID) has greatly improved currently, yet challenges remain in distinguishing the same species, such as low precision and ID switching. Inspired by the multimodal large models, we propose a Contrastive Language-Image Pretraining model for Accurate Fish Individual fine-grained Re-identification (CLIP-AFIR). Based on the cross-modal contrastive learning, a set of trainable text tokens is introduced to represent different individuals, combined with the proposed Prompt Learner Module (PLM). The text and image encoders are trained through the two-stage paradigm, which enhances the adaptability in fish recognition. To improve the ability to discriminate subtle individual differences, a lightweight Fine-grained Feature Enhancement Module (FFEM) is further designed. By using shifted windows with overlapping regions to compute mask self-attention on local areas of the image, it enables effective representation of fine-grained local variations. On the constructed grouper dataset, the proposed CLIP-AFIR shows a significant improvement in the evaluations. Applied to the non-continuous fish individual recognition task, CLIP-AFIR achieves an accuracy of 98.6%, surpassing the state-of-the-art by 5.4 points.