A screening methodology for crowdsourcing video QoE evaluation

Abstract

Recently, crowdsourcing has emerged as a cheaper and quicker alternative to traditional laboratory based Quality of Experience (QoE) evaluation for video streaming services. Crowdsourcing is a process of recruiting anonymous members of the public to solve/perform different tasks without supervision. This approach of seeking solutions from the public has been enhanced by ubiquitous internet connectivity, low cost and the ease to recruit workers without geographical restrictions. Although crowdsourcing makes it possible to save cost and reach a large group of people to perform subjective video quality testing, challenges such as validity of results and the trustworthiness of workers still remain unresolved. In this paper, we attempt to address some of these challenges by developing a screening algorithm that is able to determine the validity and trustworthiness of crowdsourcing video QoE evaluation results. This algorithm is based on evaluator's extracted data (e.g. Network IP addresses, network devices and browser information, time spent on the crowdsourcing platform, date and grading patterns). To determine the performance of this algorithm, we carried out a separate controlled laboratory based subjective video quality testing. This test enables us to determine how screened crowdsourcing results correlate with lab based results. Preliminary results show that crowdsourcing results can be improved by up to 59% when the proposed screening algorithm is applied. Moreover, the results support our assertion that crowdsourcing can provide reliable video QoE measurements if proper screening is performed on the test results.