Video and six-axis inertial sensor signals fusion for fish feeding behavior recognition in recirculating aquaculture system

Abstract

In aquaculture, real-time and accurate recognition of fish feeding behavior is of great significance to improve feeding efficiency and reduce feed waste. Multimodal approaches, such as image or motion sensors, show great potential in assessing the feeding intensity. In this paper, a fish feeding intensity classification model (VTFish) is proposed to fuse the video and six-axis inertial sensor signals. Specifically, as follows: Firstly, a Transformer is used to fusion the six-axis inertial sensor signals with video features extracted by a 3D Swin Transformer, generating feature vectors for different modalities. Then, a CAF module is introduced to guide the video features to produce fused features at low, medium, and high scales by using the time series features of the six-axis inertial sensor signal. Finally, cross-modal fusion is conducted via Cross-transformer between the high-dimensional image features and the six-axis inertial sensor's signals. Experimental results demonstrated that the proposed multi-modal VTFish model achieves a recognition accuracy of 94.29 %, which is an improvement of 3.56 % compared to the original model. Furthermore, the model outperforms single-modal recognition, with accuracy improvements of 19.29 % and 11.43 % over single image and six-axis inertial sensor modal, respectively. This model can be applied to develop intelligent feeding strategies and provide algorithm support for the development of smart feeding machines.