QoE on H.264 and H.265: Crowdsourcing versus Laboratory Testing

Abstract

Crowdsourcing as one of the newest approaches in the field of Quality of Experience (QoE) has become more and more popular. Even though many papers deal with exploring the quality, most of them use objective or subjective laboratory methods. Publications comparing the quality of any codecs using crowdsourcing method are very rare. Although some of them utilize this approach, they use one of the web-based crowdsourcing frameworks with paid crowdworkers. In this paper we decided to present our own developed web-based crowdsourcing framework and use it as unpaid approach for evaluating two well-known compression standards - H.264 and H.265 and for comparing the results getting from testing via this crowdsourcing framework with results which we got from laboratory testing. These results showed that the subjective quality testing via unpaid crowdsourcing approach can be used as a replacement to the subjective laboratory testing, especially in cases where not very sophisticated evaluation is required.