Observation and Visualization of Subjectivity-based Annotation Tasks

Annotation is an upstream process for constructing training data for machine learning tasks. The reliability of annotation is very important for the reliability of machine learning. The annotations vary from worker to worker, and differences in these tendencies may impair the reliability of the data. This is especially relevant for tasks that depend on the subjectivity of the workers. This study aims to realize reliable annotation by observing the annotation results of workers. As a specific example, we applied the annotations of three workers who evaluated facial expressions by the Likert scale on 977 face images as a subject. We verified the reliability of the annotations from the visualization results.