RC-UMT: Multimodal fish feeding intensity classification network with enhanced feature extraction capability in aquaculture

Abstract

Quantifying fish feeding intensity in real-time is crucial for devising scientific feeding strategies and improving production efficiency. Unimodal fish feeding intensity recognition tasks, such as audio or video, often fail to fully reflect the global characteristics, and lead to unreliable results. To address the limitations, a video and audio fusion model Reinforced Cross-modal Uni-Modal Teacher (RC-UMT) is introduced to evaluate fish feeding intensity classification, which can categorise feeding intensity into four different categories. The specific implementation is as follows: Firstly, the backbone in the original Uni-Modal Teacher (UMT) model is replaced with a Res2Net network, by weighting spatial coordinates of input feature maps, the Coordinate Attention (CA) mechanism in the Res2Net network enhances the ability to capture key multi-frequency features in audio signals. Secondly, RepViT, a lightweight convolutional network inspired by Vision Transformer principles, is introduced to the UMT, which can extract both global and local visual features simultaneously, thereby improving multi-level semantic representation. Finally, an affine transformation matrix with manifold preservation is introduced to enhance the fusion process of the original UMT, which preserves the manifold structure of original modal features, thereby achieving more efficient cross-modal fusion. Results from the experimentation indicate that the RC-UMT model achieves a classification accuracy of 93 %, outperforming the original UMT model by 7 % while reducing parameters by 25.41 %. Compared to audio-only and video-only modalities, the accuracy improves by 7 % and 6.5 %, respectively. Therefore, the proposed video-audio multimodal model enables real-time, high-precision feeding intensity classification, providing technical support for improving smart feeding equipment.