Evaluating Annotators Consistency with the Aid of an Innovative Database Schema

Abstract

Automatic semantic tagging of multimedia is still inefficient due to the difficulties in modelling abstract or complex terms using low level features. The degree of consensus and homogeneity in judgements among annotators is very important in semantic image and video retrieval. In this paper we present a novel method in evaluating the annotators consistency, which uses an innovative database schema and combines two different annotation approaches. A set of 100 images were annotated by 16 annotators using vocabulary keywords and free keywords. The results indicate that combination of annotation methods may lead to increased annotation consistency compared to a single method but this is not a general fact. As expected the use of free keywords and images require tagging that is not directly related to their content, lead to increase the annotators inconsistency.