Research on Multimodal Recognition Methods for Perimeter Security Based on the Fusion of DVS and Video Surveillance

Perimeter security systems widely employ distributed fiber-optic sensing technology and video surveillance as sensing means. However, significant limitations remain in practical applications. Distributed fiber-optic sensing technology is susceptible to interference from environmental noise coupling, resulting in a high false alarm rate. Meanwhile, video surveillance technology faces issues such as increased image noise and blurred target outlines in complex environments. These problems are compounded by the complexity of the background, which makes it difficult to accurately identify subtle behavioral differences. To address these challenges, this article proposes a multimodal fusion classification model, HMFusionNet, which leverages the complementary information from distributed vibration sensing (DVS) and video surveillance to improve classification accuracy. First, we introduce the CGANet module to extract features from 1-D fiber vibration signals and capture the periodic characteristics of the fiber time series. Second, we design the PoseMobiNet module to extract 2-D image features based on human keypoint data and RGB image information, addressing the complexities of the perimeter security background and the subtleties of behavioral differences among intruders. During the feature fusion stage, we propose a probabilistic weighting-based late fusion strategy to integrate decision-level information from both modalities. Finally, using a multimodal dataset constructed based on a real-world perimeter security scenario, the HMFusionNet achieves a detection accuracy of 97.7% with a recognition time of less than 0.1 s.