Performance Issues in Parallelizing Data-Intensive Applications on a Multi-core Cluster

Abstract

The deluge of available data for analysis demands the need to scale the performance of data mining implementations. With the current architectural trends, one of the major challenges today is achieving programmability and performance for data mining applications on multi-core machines and cluster of multi-core machines. To address this problem, we have been developing a runtime framework, FREERIDE, that enables  parallel execution of data mining  and data analysis tasks.The contributions of this paper are two-fold: 1) This paper describes and evaluates various shared-memory parallelization techniques developed in our run-time system on a cluster of multi-cores, and  2) We report on a detailed performance study to understand why certain parallelization techniques out-perform othertechniques for a particular application.