VFE: A large-scale video future event description dataset for evaluating video temporal prediction

Given a video, humans can predict subsequent events in the video and generate reasonable descriptions based on the acquired information and prior knowledge. This ability requires in-depth analysis of dynamic visual information in videos and the comprehensive use of extensive world knowledge for logical reasoning and prediction. However, current visual systems have not yet reached a satisfactory level regarding similar temporal prediction capability. To evaluate this new application, we construct a dataset called VFE (Video Future Event Description), a large-scale dataset for subsequent video event prediction. The VFE dataset contains over 84K video clips, and each clip is equipped with a video and description of the premise event and a predicted description of the subsequent events. To evaluate video temporal prediction, we propose a task, video future event prediction, to generate possible future event descriptions for subsequent unseen video clips based on the premise video. In this paper, we also propose a baseline model for evaluating the VFE dataset. The experimental results indicate the challenge of this task, and the ability of the visual system in complex video temporal prediction needs to be further explored. The dataset and code are available at https://github.com/keyancaigou/VFE.