Towards a Scene-Based Video Annotation Framework

Abstract

The amount of video in the web is huge and searching specific part of a video can be achieved using a content or text based video indexing, grouping, searching and retrieval approaches. To realize content based video searching, in this work we propose scene based video annotation for the identification and labeling of events and objects in a video with a descriptive text. Video annotation requires a knowledge base to define semantic meaning of events and objects in the video. Manual and semi-supervised video annotation approaches fail as both require expertise for the correct identification and labeling of video concepts. Annotation requires a great deal of concept dependency and relatedness processing to give descriptive statement to a scene in the video. This paper introduces a novel scene based video annotation framework to provide scene level semantic description of videos. The framework uses audio component of the scene to support event and object identification and has proper filtering and normalization. The framework provides concept relatedness, concept formulation, shot and scene level video annotations. To validate the capability of the proposed framework, we developed a prototype that shows a scene level video annotation. The framework is evaluated using a standard video processing evaluation dataset for its accuracy in event and object prediction, and the overall accuracy and usability of the system is evaluated using human ratings. The proposed approach exhibits 81% accuracy in object and event prediction and an average user rating of 3.47 out of 4 in overall system evaluation.