Multimodal sow lameness classification method integrating spatiotemporal features

Abstract

Sow lameness may result in reduced swine farming efficiency, decreased production performance, and diminished economic profitability of farms. Therefore, the automatic and accurate prediction of sow lameness is crucial for enhancing health monitoring systems and improving farm profitability. This paper introduces a Contour and Skeleton Fusion-based Multimodal Network (CSF-MN) for classifying the severity of sow lameness. The Contour Feature Classification (CFC) module within the CSF-MN framework employs the FYOLOv8s-Seg algorithm to extract contour features of sows, which are then processed by the SimTSM algorithm to train a contour classification model. Meanwhile, the Skeleton Feature Classification (SFC) module uses the FYOLOv8s-Pose algorithm for skeletal feature extraction and integrates the NLPoseC3D algorithm to train a skeletal classification model. To detect lameness, prediction confidences from both models are dynamically fused using a weight assignment mechanism. To validate the effectiveness of the method, 321 samples are randomly selected from a total of 459 samples for K-fold cross-validation. The 321 samples are divided into 10 subsets, with 8 subsets used as the training set and the remaining 2 subsets used as the validation set in each iteration. This process is repeated 10 times, and the results from all 10 iterations are used to evaluate the performance. Experimental results demonstrate that the CSF-MN network achieved an accuracy of 94.2 %, specificity of 96.8 %, and sensitivity of 97.4 % on the test set. These results indicate that the proposed approach effectively integrates spatiotemporal features from sow gait, enabling an accurate assessment of lameness severity.