Visualization of Individual Variation of Multiple Annotators Working on Training Datasets for Machine Learning

Abstract

Quality of training datasets is essential for the quality of machine learning. Machine learning projects often invite multiple workers for these annotation tasks for training dataset creation. It is important to observe on what types of contents multiple workers make different annotations, or which workers often make abnormal annotations, to guarantee the quality of training datasets. This paper presents a tool for the visualization of abnormality of annotations by multiple workers. The tool generates a matrix of abnormality of annotations for each of the images by each of the workers and displays as a heatmap. This paper introduces an example using a training dataset where estimated ages are annotated to 7,748 pictures of human faces by eight workers.