Distributed web spider based on cluster and grid technologies

Abstract

We present our web spider developed as a part of a web search engine with language grammar support. Spider is optimized for distributed memory multi- processor arhitecture, with multithreading core processors. As a part of this work, distributed data models have been devel- oped to optimize fast manipulation and long time storage of the large amount of data generated by process of web indexing. Development of the spider is part of CRO- GRID Applications project, subproject of the CRO-GRID project.