Building crawler engine on cloud computing infrastructure

Abstract

This paper is aimed to create implementation crawler engine or search engine using cloud computing infrastructure. This approach use virtual machines on a cloud computing infrastructure to run service engine crawlers and also for application servers. Based on our initial experiments, this research has successfully built crawler engine that runs on Virtual Machine (VM) of cloud computing infrastructure. The use of Virtual Machine (VM) on this architecture will help to ease setup or installation, maintenance or VM terminating that has been running with some particular service crawler engine as needed. With this infrastructure, the increasing or decreasing in capacity and capability of multiple engine crawlers could set easily and more efficiently.