A survey on dynamic scene understanding using temporal knowledge graphs: From scene knowledge representation to extrapolation

Abstract

Dynamic scene understanding is the process of extracting information from video, identifying and inferring entities and relations within the scene, with the aim of thoroughly analyzing complex scenes that evolve over time. This process leverages temporal knowledge graphs to achieve a deep and comprehensive understanding of dynamic environments and is widely applied in areas such as autonomous driving, surveillance, and video analysis. Initially, scene knowledge representation is explored as the foundational step in dynamic scene understanding, achieved through the generation of temporal knowledge graphs. These graphs are categorized based on temporal granularity. Temporal knowledge graphs are divided into multiple-frame dynamic graphs and single-frame dynamic graphs. The generation methods for multiple-frame dynamic graphs are categorized into fragment-based and sliding-window approaches, while single-frame dynamic graphs primarily utilize transformer-based methods. This section provides an overview of the generation models for temporal knowledge graphs. Subsequently, dynamic scenes are further analyzed using extrapolation methods, which are classified into entity-based and relation-based modeling approaches. Entity-based modeling methods mainly include temporal point processes and graph neural network techniques, while relation-based modeling focuses on reinforcement learning and meta-learning techniques. This section summarizes various existing extrapolation techniques within these categories. Finally, the paper discusses the challenges associated with temporal knowledge graphs and explores potential research directions, offering insights into future advancements in dynamic scene understanding.