Evaluating and improving integration quality for heterogeneous data sources using statistical analysis

Abstract

This paper considers the problem of integrating heterogeneous semi-structured data sources with the purpose of estimating integration quality (IQ). Integration of such data sources leads to results with unpredictable trustworthiness and none of the existing methods is capable of accounting for the uncertainty which is accumulated over all of the integration steps and which affects integration quality. To compute the uncertainties we suggest using a well-established statistical method Latent Class Analysis (LCA). This method allows to analyze the influence of the latent factors associated with the real-world entities on the set of data. We show on examples how the proposed approach can be used for evaluating and improving IQ giving an important tool to the users concerned with the data's trustworthiness.