XHAMI -- Extended HDFS and MapReduce Interface for Image Processing Applications

Abstract

Hadoop Distributed File System (HDFS) and MapReduce model have become de facto standard for large scale data organization and analysis. Existing model of data organization and processing in Hadoop using HDFS and MapReduce are ideally tailored for search and data parallel applications, for which there is no data dependency with neighboring/adjacent data. Many scientific applications such as image mining, data mining, knowledge data mining, satellite image processing etc., are dependent on adjacent data for processing and analysis. In this paper, we discuss the requirements of the overlapped data organization and propose XHAMI as a two phase extensions to HDFS and MapReduce programming model to address such requirements. We present the APIs and discuss their implementation specific to Image Processing (IP) domain in detail, followed by sample case studies of image processing functions along with the results. XHAMI though has little overheads in data storage and input/output operations, but greatly improves the system performance and simplifies the application development process. The proposed system works without any changes for the existing MapReduce models with zero overheads, and can be used for many domain specific applications where there is a requirement of overlapped data.