Early stroke behavior detection based on improved video masked autoencoders for potential patients

Stroke is the prevalent cerebrovascular disease characterized by significant incidence and disability rates. To enhance the early perceive and detection of potential stroke patients, the early stroke behavior detection based on improved Video Masked Autoencoders (VideoMAE) for potential patients (EPBR-PS) is proposed. The proposed method begins with novel time interval-based sampling strategy, capturing video frame sequences enriched with sparse motion features. On the basis of establishing the masking mechanism for adjacent frames and pixel blocks within these sequences, The EPBR-PS employes pipeline mask strategy to extract spatiotemporal features effectively. Then, the local convolution attention mechanism is designed to capture local dynamic feature information, and central to the EPBR-PS is the integration of local convolutional attention mechanism with VideoMAE's multi-head attention mechanism. This integration facilitates the simultaneous leveraging of global high-level semantics and local dynamic feature information. Dual attention mechanism-based method for the fusion of these global and local features is proposed. After that, the optimal parameters of EPBR-PS were determined through the experiment of learning rate and fusion weights of different features. On the NTU-ST dataset, comparative analysis with eight models demonstrated the superiority of EPBR-PS, evidenced by the average recognition accuracy of 89.61%, surpassing that 1.67% over the benchmark VideoMAE. On the HMDB51 dataset, EPBR-PS has Top1 of 71.31%, which is 0.73% higher than that of the VideoMAE, providing the viable behavior detection for perception early signs of potential stroke in the home environment. This code is available at https://github.com/wang-325/EPBR-PS/.