State Summarization of Video Streams for Spatiotemporal Query Matching in Complex Event Processing

Abstract

Modelling complex events in unstructured data like videos not only requires detecting objects but also the spatiotemporal relationships among objects. Complex Event Processing (CEP) systems discretize continuous streams into fixed batches using windows and apply operators over these batches to detect patterns in real-time. To this end, we apply CEP techniques over video streams to identify spatiotemporal patterns by capturing window state. This work introduces a novel problem where an input video stream is converted to a stream of graphs which are aggregated to a single graph over a given state. Incoming video frames are converted to a timestamped Video Event Knowledge Graph (VEKG) [1] that maps objects to nodes and captures spatiotemporal relationships among object nodes. Objects coexist across multiple frames which leads to the creation of redundant nodes and edges at different time instances that results in high memory usage. There is a need for expressive and storage efficient graph model which can summarize graph streams in a single view. We propose Event Aggregated Graph (EAG), a summarized graph representation of VEKG streams over a given state. EAG captures different spatiotemporal relationships among objects using an Event Adjacency Matrix without replicating the nodes and edges across time instances. These enable the CEP system to process multiple continuous queries and perform frequent spatiotemporal pattern matching computations over a single summarised graph. Initial experiments show EAG takes 68.35% and 28.9% less space compared to baseline and state of the art graph summarization method respectively. EAG takes 5X less search time to detect pattern as compare to VEKG stream.