An Interactive Image Retrieval Approach to Searching for Images on Social Media

Searching for images posted within social media services such as Twitter relies on matching textual queries to the contents of the posts that include the images. Unfortunately, social media posts may not always provide accurate or meaningful descriptions of the contents of the embedded images, making searching for images a challenging task. In this research, we augment the textual contents of the posts with new information extracted from the images using image processing and deep learning methods, and provide a visual interface to enable interactive image retrieval. A user study was conducted with 28 participants to collect evidence on how our approach was used in relation to Vakkari's three-stage model of information seeking. We also analyzed participants' perceptions of usefulness, ease of use, and satisfaction in comparison to a common grid-based image search interface. The results from this study highlight the value of providing visual and interactive features to enable searchers to discover images from social media sources.