Incorporating the Graph Representation of Video and Text into Video Captioning

Abstract

Video captioning is to translate the video content into the textual descriptions. In the encoding phase, the existing approaches encode the irrelevant background and uncorrelated visual objects into visual features. That leads to semantic aberration between the visual features and the expected textual caption. In the decoding phase, the word-by-word prediction infers the next word only from the previously generated caption. That local text context is insufficient for word prediction. To tackle the above two issues, the representations of video and text stem from the convolution on two graphs. The convolution on the video graph distills the visual features by filtering the irrelevant background and uncorrelated salient objects. The key issue is to figure out the similar videos according to the video semantic feature. The word graph is constructed to help incorporate global neighborhood among words into word representation. That word global neigh-borhood serves as the global text context and compensates the local text context. Results on two benchmark datasets show the advantage of the proposed method. Experimental analysis is also conducted to verify the effectiveness of the proposed modules.