Multimodal knowledge distillation framework for fish feeding behaviour recognition in industrial aquaculture

Abstract

Fish feeding behaviour recognition based on machine vision has great significance in industrial aquaculture. Due to the problems of turbid water and overlapping fish during the feeding stage, accurate and low-cost feeding behaviour recognition becomes challenging in actual industrial aquaculture. To address these issues, a novel Multimodal Knowledge Distillation Recognition (MMKDR) framework, based on multimodal fusion and enhanced knowledge distillation, is proposed, to achieve low-complexity and low-cost deployment. Specifically, we utilised the Feature Extraction module of ConvNeXt-T (CNXFE) to extract image features from video streaming. Then, an Improved Multimodal Fusion (IMF) module is designed to generate the fused feature, which can dynamically adjust the weights of the image and water quality features. Next, a Lightweight Feeding Intensity Classification (LFIC) module is designed to predict the fish feeding intensity from the fused feature, which helps to optimise feeding strategies and reduce aquaculture management cost. To deploy the student model on low-cost embedded devices, we further reduce the parameters of the CNXFE and IMF, and obtain the smaller student model with 2.49M parameters. An Enhanced Knowledge Distillation (EKD) scheme, with semi-supervised domain adaptation, is present to achieve knowledge transfer with better recognition accuracy. It can reduce devices and data annotation costs to promote low-cost and low-carbon aquaculture. We carried out experiments and evaluated MMKDR in a real industrial aquaculture environment. The results demonstrated that the student model achieved an accuracy of 96.65 % on the testing set, and an accuracy of 91.36 % using low-cost embedded device in real industrial aquaculture scenarios.