Visual Attention based Cognitive Informative Frame Extraction Method for Smart Crowd Surveillance

Abstract

In a smart surveillance system, the amount of video data has increased exponentially due to the increase in the number of monitoring devices and IoT sensors. To make smart and real-time decisions without latency in communication from these voluminous data is a tedious task. In this context, selecting informative frames from the video is of great importance that helps to extract only the salient features for further processing without latency and bandwidth constraints. In this paper, we are proposing a fast and reliable method for selecting informative frames from video sequences based on the human cognition process of visual attention to preserve the Spatio-temporal properties of the video. The proposed method extracts the informative frames using the frame informative score calculated based on visual attention maps, superpixel segmentation, and temporal information. Since our purpose is for analyzing crowd behavior from video data in a smart environment, we take two publicly available crowd video datasets for our experiments. The results show that the proposed approach is successful in extracting relevant video frames in linear time by preserving their spatial and temporal properties. We also analyze the feasibility of the proposed method in a fog computing-based simulated IoT framework, and it has been verified that the proposed cognitive approach could efficiently address the concerns of latency and bandwidth in smart surveillance environments.