Implementation and Performance Evaluation of a Hybrid Distributed System for Storing and Processing Images from the Web

Multimedia applications have undergone tremendous changes in the recent past that they have called for a scalable and reliable processing and storage framework. Image processing algorithms such as pornographic content detection becomes a lot more challenging in terms of accuracy, recall, and speed when run on billions of images. This paper presents the design and implementation of a hybrid-distributed architecture that uses Hadoop distributed file system for storage and Map/Reduce paradigm for processing images, crawled from the web. This architecture combines the power of Hadoop framework when there is a need to parallelize the task, as Map/Reduce jobs and uses stand alone crawler nodes to fetch relevant contents from the web. Evaluations on real world web data indicate that the system can store and process billions of images in few hours.