Assessing the quality of a Consensus determined using a multi-level approach

Abstract

The following paper investigates a multilevel approach to data integration using the widely accepted Consensus Theory. We focus on an issue related to an initial classification of raw input data into groups that can be integrated in parallel. A final consensus is a result of the integration of obtained partial outcomes. Our main research concerns an application of Fleiss' kappa value, which in the literature is a well known measure that describes how consonant the data in a selected set are. In other words - for a given set of values, the higher the value of this measure, the higher its inner consistency. Therefore, we have attempted to answer the question whether or not the initial data should be divided into coherent groups or into highly divergent subsets, that better represent the whole input. We present a theoretical background, broad description of a series of experiments that we have performed and their statistical analysis.