Examining subjective evaluation methods used in multimedia Quality of Experience research

Abstract

Many different evaluation methods for assessing the Quality of Experience (QoE) of multimedia systems nowadays exist in official QoE recommendations and methodological publications as well as related areas of research like User experience (UX) and usability. Although the goal of each method is to evaluate QoE in user studies either quantitatively or qualitatively, our first approach to classify these methods beyond results is complex. In this paper, we extend the previously developed comparison model, which was based on literature review only with an evaluation of the developed classes by expert interviews. The interviews deepened our understanding of method comparison and guided method selection process. The results show that different levels of importance can be assigned to the comparison criteria of the first model. Based on the experts' contributions in the interview, we propose a revised comparison model, which is more suitable for adaptation towards its operationalization and the further comparison tool development.