An Application of Distributed Data Mining to Identify Data Quality Problems

Abstract

When dealing with huge data sets, during the integration process of distributed data into a single data warehouse, one is not only confronted with time and security factors but with the well known problem of low data quality as well. In order to cope with such issues that the integration of distributed data often is faced with, we present in this paper an approach that applies distributed data mining, to facilitate a data quality analysis of the data in their distributed state. Data quality problems are identified by a classifier, which uses the knowledge gained from the clustering (subspace clustering) process performed on the distributed data. Experiments on real data show that the distributed analysis results are comparable to those conducted on the central data warehouse using classical data mining.