Simultaneous exploitation of explicit and implicit tags in affect-based multimedia retrieval

Affect-based retrieval of multimedia items requires tags that describe the content of these items. These tags are added by users that interact with the items. In this paper, it is shown to what extent different ways of creating the tags result in similar or non-similar information about the item. Three types of affective tags are being considered here: explicit self-assessed tags, implicit multimedia-based tags, and implicit physiology-based tags. The novelty of this paper is to show that affect-based retrieval accuracy (as measured by precision and recall) benefits from having a database that contains both explicit and implicit tags. A database that contains a mixture of explicit and implicit tags has higher retrieval accuracy when compared to a database that contains uniquely either explicit or implicit tags. This shows that information in explicit and implicit tags is complementary rather than redundant. The improvement in retrieval accuracy is immediately evident when explicit tags are added. Results for low recall rates are of particular importance because these focus on the most relevant items in a database. When over 60% of the items in the database with implicit tags are also tagged by explicit tags, retrieval accuracy at a low recall rate (0.1) is higher than accuracy based uniquely on explicit tags.