An emotion-aware search engine for multimedia content based on deep learning algorithms

Abstract

Nowadays, large amounts of unstructured data are available online. Such data often contain users' emotions and feelings about a variety of topics but their retrieval and selection on the basis of an emotional perspective are usually unfeasible through traditional search engines, which only rank web content according to its relevance with respect to a given search keyword. For this reason, in the present work we introduce the architecture of a novel emotion-aware search engine that can return search results ranked on the basis of seven human emotions. Using this system, users can benefit from a more advanced semantic search that also takes into account emotions. The system uses emotion recognition algorithms based on deep learning to extract emotion vectors from texts, images and videos and then populates an emotional index to allow users to visualise results related to given emotions. We also discuss and evaluate different deep learning models for building emotional indexes from texts, images and videos.