Infusing perceptual expertise and domain knowledge into a human-centered image retrieval system: a prototype application

Abstract

Traditional content-based image retrieval techniques, which primarily rely on image content at the pixel level, are not effective in accessing images at the semantic level. Defining approaches to incorporate experts' perceptual and conceptual capabilities of image understanding in their domain of expertise into the retrieval processes promises to help bridge this semantic gap. Towards accomplishing this, we design and implement a novel multimodal interactive system for image retrieval. To incorporate human expertise, the system stores expert-derived information extracted from two human sensor modalities that intuitively relate to image search, eye movements and verbal descriptions, both generated by medical experts. Experimental evaluation of the system shows that by transferring experts' perceptual expertise and domain knowledge into image-based computational procedures, our system can take advantage of the different human-centered modalities' respective strengths and improve the retrieval performance over just using image-based features.