Parallelization of Vertical Search Engine using Hadoop and MapReduce

Abstract

In this paper, we build a parallelized Vertical Search Engine on Apache Hadoop cluster. Domain of our Vertical Search Engine is Computer related terminologies and it takes seed URLs of computer domain extracted from Wikipedia. These web-pages are then crawled and parsed with the help of Apache Nutch crawler and stored into Apache HBase. Linguistic processing like removal of stop words and stemming are performed on the content of the web pages stored in the database. Inverted index is constructed and results are ranked as per the ranking algorithm described later in the paper. This paper mainly focuses on obtaining the most relevant results in shortest time possible by distributed processing.