Towards training naïve participants for a perceptual annotation task designed for experts

Abstract

Technical Causes Analysis (P.TCA) is a method for identifying technical causes of sub-optimum speech transmission quality. Originally created as an expert procedure for the annotation of speech samples, its applicability to naïve listener was also studied. Due to the low agreement of naïve listener annotations, it was suggested that detailed training methods are necessary to lift naïve annotations to an agreement level of experts. The aim of this work was to develop training methods for naïve annotators. For this, two different training procedures were developed and tested in two separate annotation experiments. The results are analyzed and discussed regarding the effects of the trainings and their implications for the P.TCA annotation scheme. The outcome shows that these training methods did not meet the expectations for improving the inter-rater agreement of naïve annotators. It is concluded that trainings of 15 to 20 minutes rather confuse naïve annotators by conveying too much information in too little time, and that they are not sufficient to prepare naïve annotators. It is argued that much more extensive training is needed to raise naïve annotators to expert level, and that such a training must include both, in-depth introduction to the annotation process as well as detailed presentation and exercise regarding the P.TCA degradations.