Effect of Number of Stimuli on Users Perception of Different Speech Degradations. A Crowdsourcing Case Study

Crowdsourcing (CS) has established as a powerful tool to collect human input for data acquisition and labeling. However, it remains the question about the validity of the data collected in a CS platform. Sometimes, the users work carelessly or they try to tweak the system to maximize their profits. This paper reports on whether the number of speech stimuli presented to the listeners has an impact on the user perception of certain degradation conditions applied to the speech signal. To this end, a crowdsourcing study has been conducted with 209 listeners that were divided in three non-overlapping user groups, each of which was presented with tasks containing a different number of stimuli: 10, 20, or 40. Listeners were asked to rate speech stimuli with respect to their overall quality and the ratings were collected on a 5-point scale in accordance with ITU-T Rec. P.800. Workers assessed the speech stimuli of the database 501 from ITU-T Rec. P.863. Additionally, the influence of certain speech signal characteristics, such as interruptions and bandwidth, on the quality perception of the workers was investigated.