DEDGraph: Delay Embedding of Dynamic Graph for Temporal Action Segmentation

Abstract

In real-world interactive applications, where videos are generated in real-time and require immediate feedback, online segmentation has practical advantages over offline inference. Many excellent previous models have been developed for offline scenarios, while real-time prediction for temporal action segmentation (TAS) is a difficult task. Some interactive applications can tolerate a certain amount of delay. In this paper, we propose a node delay embedding of a dynamic graph for real-time TAS. We transform the video stream into a dynamic graph stream that evolves over time. We define past, current, and future nodes to construct sub-graphs at each step. Specifically, future nodes are sampled using our proposed node delay method. A graph model is utilized to aggregate past, current, and future node information to update the representation of current nodes and predict their labels. To the best of our knowledge, it is the first real-time TAS graph model with delay embedding. Experiments show that delay embedding enhances node representation and improves performance. Overall, our proposed approach provides a promising solution for real-time TAS.