Accuracy of categorical measurements: Nominal scale

Abstract

Classification accuracy, considered from a categorical measurement point of view, is particularly crucial in scenarios where the cost of false output is high. Categorical measurement means that the object’s property under study is presented on a scale consisting of K exclusive classes/categories, forming a comprehensive spectrum of this property. Often such classifications are provided by a group of measurers/classifiers participating in collaborative study (e.g. inter-laboratory or inter-operator comparisons, proficiency testing projects, comparison of different methods or algorithms etc.). Previously, only the metrological properties of the fixed-factor model were studied. A random factors statistical model for analyzing accuracy from collaborative studies is presented in this paper. We assume that due to measurement/classification errors, a property belonging to category i will be classified by a collaborator as category k with probabilities $p_{k\left(i\right)}$ (confusion matrix), distributed between collaborators according to the Dirichlet distribution for every given i, whereas category counts of repeated classifications by every collaborator are distributed according to corresponding multinomial distribution. We propose unbiased estimators for repeatability and classifiers’ components of the total precision as well as trueness, based on total categorical variation decomposition and the distance from an ideal classification metrics. We discuss possible options for statistical homogeneity/heterogeneity tests. In the framework of the proposed model, the issue of assessing concordance and discordance between classifiers is also discussed.