{"title":"“A Crowdsourcing-based QoE Evaluation of an Immersive VR Autonomous Driving Experience”","authors":"Guilherme Daniel Gomes, R. Flynn, Niall Murray","doi":"10.1109/QoMEX51781.2021.9465390","DOIUrl":"https://doi.org/10.1109/QoMEX51781.2021.9465390","url":null,"abstract":"Due to COVID-19, crowdsourcing has gained momentum as an alternative methodology for continuing research and for Quality of Experience (QoE) assessment. Employing this approach, we remotely evaluated the user perceived QoE of two different visual rendering formats as part of an Autonomous Vehicles (AVs) simulation. The aim was to investigate the participant’s QoE when testing AV technology in distinct visual rendering qualities (lowpoly vs high-poly) of an online streamed 360° car riding experience. In addition, a scoring model based on the expected reliability of each level of the remote assessment was designed. Findings suggest that the consumer’s preferences towards the adoption of AV technology is highly determined by the system and human effects on Influence Factors (IFs). Moreover, the adequacy of reliability into a mathematical model is highlighted as a potential turning point for QoE assessment, by carrying out the evaluation tasks from the laboratory environment into the internet, particularly relevant in pandemic times.","PeriodicalId":160378,"journal":{"name":"2021 13th International Conference on Quality of Multimedia Experience (QoMEX)","volume":"86 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116705053","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Rafael Zequeira Jiménez, Gabriel Mittag, S. Möller
{"title":"Removing the Bias in Speech Quality Scores Collected in Noisy Crowdsourcing Environments","authors":"Rafael Zequeira Jiménez, Gabriel Mittag, S. Möller","doi":"10.1109/QoMEX51781.2021.9465442","DOIUrl":"https://doi.org/10.1109/QoMEX51781.2021.9465442","url":null,"abstract":"Subjective speech quality scores needed to train models for the automatic evaluation of telecommunication systems have generally been collected by conducting demanding laboratory tests. Alternatively, crowdsourcing has emerged as a valid method to address user-centered studies to a large pool of users over the Internet. However, crowdsourcing users often do not follow the instructions and may execute the assigned task in noisy environments. The validity of the data collected in a disturbed environment is questionable, especially in speech quality assessment and other audio-related tasks. This work investigates the applicability of different ensemble-based and non-linear models to correct the bias found in speech quality ratings given to a German speech dataset in noisy crowdsourcing environments. Such a model would help to avoid throwing away quality scores that could be used. The model was trained with data collected in a speech quality assessment study conducted in a simulated crowdsourcing environment in the laboratory. Two groups of listeners rated the quality of speech stimuli in the presence of environmental noise at different levels. The noise under test was street traffic, and the levels ranged from 36dBA to 65. 5dBA. A fine-tuned gradient boosting regressor yielded the best results with a R2 score of 0.90 and RMSE of 0.416.","PeriodicalId":160378,"journal":{"name":"2021 13th International Conference on Quality of Multimedia Experience (QoMEX)","volume":"29 2","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114108301","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ailbhe Gill, Emin Zerman, Martin Alain, Mikael Le Pendu, A. Smolic
{"title":"Focus Guided Light Field Saliency Estimation","authors":"Ailbhe Gill, Emin Zerman, Martin Alain, Mikael Le Pendu, A. Smolic","doi":"10.1109/QoMEX51781.2021.9465428","DOIUrl":"https://doi.org/10.1109/QoMEX51781.2021.9465428","url":null,"abstract":"Light field imaging enables us to capture all light rays in a visual scene. As light fields are four-dimensional, their captures come with an increased amount of information to take advantage of. This has stimulated ongoing light field specific research into virtual viewpoints and shallow depth of field rendering, commonly called refocusing. However, the computation time and memory required to perform these operations can make tasks such as real-time rendering impractical. One solution is to exploit the salient information of light fields to focus resources on regions that attract visual attention when using these algorithms. Although saliency estimation methods for light fields have been previously explored, these focus mainly on salient object segmentation with the goal of generating one saliency map per light field.Aiming to create a basis for a 4D saliency prediction model analogous to light fields, this paper proposes a saliency estimation method specific to light fields that considers the refocusing operation. The proposed method modifies an existing view rendering algorithm with focus guidance, obtained from the light field disparity. This facilitates the construction of saliency maps without the need to render the corresponding view itself, which will help to speed up processing operations that are compatible. The results show that the proposed saliency estimation approach yields very good predictions of visual attention across multiple planes of the light field. We anticipate that this approach can be extended for a range of rendering applications.","PeriodicalId":160378,"journal":{"name":"2021 13th International Conference on Quality of Multimedia Experience (QoMEX)","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123677025","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Fatima Boudjerida, A. Lahoulou, T. Falk, Z. Akhtar
{"title":"Parametric Audio-visual Quality Measurement Based on Interpretable Fuzzy Logic","authors":"Fatima Boudjerida, A. Lahoulou, T. Falk, Z. Akhtar","doi":"10.1109/QoMEX51781.2021.9465453","DOIUrl":"https://doi.org/10.1109/QoMEX51781.2021.9465453","url":null,"abstract":"In this paper, a fuzzy inference based system for the quality of experience assessment is presented. In particular, two models have been proposed to assess the perceived quality of experience. The first one is a global model which estimates the overall audiovisual quality without going through the individual evaluations of visual and auditory modalities while the other model has been created by merging fuzzy inference systems based on separate auditory and visual objective quality scores. Two different sets of parameters have been tested for the second model leading to two different measures. The contribution of this research lies in the application of the fuzzy inference logic to estimate the audiovisual quality of multimedia data. The experimental results on a publicly available quality dataset show competitive predictive performances of the proposed measures when compared to two existing audiovisual metrics based on random forests regression and multilayer perceptron.","PeriodicalId":160378,"journal":{"name":"2021 13th International Conference on Quality of Multimedia Experience (QoMEX)","volume":"55 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123817068","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Florian Metzger, Norman Stulier, Kathrin Borchert, Matthias Hirth
{"title":"Relationship Status: It’s Complicated. Using APM as a QoE-qualifying Tempo Metric","authors":"Florian Metzger, Norman Stulier, Kathrin Borchert, Matthias Hirth","doi":"10.1109/QoMEX51781.2021.9465396","DOIUrl":"https://doi.org/10.1109/QoMEX51781.2021.9465396","url":null,"abstract":"The results of studies investigating the perceived quality of different video games in various environments are often not easy to compare. While one input delay value might be perfectly fine for one game, it might completely break immersion and flow in another. Typically, one would assume that video games can be categorized based on their genre to reach a certain comparability, but that has not proven true in the past. Thus, efforts are made to find new ways of categorization for this issue that are based on objective and quantifiable metrics. In this work, we focus on the merits of Actions per Minute (APM) as one such candidate metric as there is already preliminary usage of the metric for this purpose in the literature. We explore a diverse dataset of real-world DOTA 2 matches and try to understand the characteristics and stability of APM and its relationship to other game-centric and player-centric objective metrics. This is an initial attempt to understand how stable such a metric would be even for just one game in the larger quest to find a suitable cross-game categorization metric. Our exploration has uncovered player-specific influences in the APM. But we are confident that with a narrow, filtered definition of the APM this approach can be used to define APM ranges that resemble the tempo of a game to be used as a basis to classify games for Quality of Experience research.","PeriodicalId":160378,"journal":{"name":"2021 13th International Conference on Quality of Multimedia Experience (QoMEX)","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131441068","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Michela Testolina, Evgeniy Upenik, J. Ascenso, F. Pereira, T. Ebrahimi
{"title":"Performance Evaluation of Objective Image Quality Metrics on Conventional and Learning-Based Compression Artifacts","authors":"Michela Testolina, Evgeniy Upenik, J. Ascenso, F. Pereira, T. Ebrahimi","doi":"10.1109/QoMEX51781.2021.9465445","DOIUrl":"https://doi.org/10.1109/QoMEX51781.2021.9465445","url":null,"abstract":"Lossy image compression is a popular, simple and effective solution to reduce the amount of data representing digital pictures. In most lossy compression methods, the reduced volume of data in bits is achieved at the expense of introducing visual artifacts in the picture. The perceptual quality impact of such artifacts can be assessed with expensive and time-consuming subjective image quality experiments or through objective image quality metrics. However, the faster and less resource demanding objective quality metrics are not always able to reliably predict the quality as perceived by human observers. In this paper, the performance of 14 objective image quality metrics is benchmarked against a dataset of compressed images labeled with their subjective quality scores. Moreover, the performance of the above objective quality metrics in predicting the subjective quality of images distorted by both conventional and learning-based lossy compression artifacts is assessed and conclusions are drawn.","PeriodicalId":160378,"journal":{"name":"2021 13th International Conference on Quality of Multimedia Experience (QoMEX)","volume":"14 3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126261736","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Marta Orduna, Jesús Gutiérrez, Carlos Manzano, David Ruiz, J. Cabrera, César Díaz, P. Pérez, Narciso García
{"title":"EVENT-CLASS: Dataset of events in the classroom","authors":"Marta Orduna, Jesús Gutiérrez, Carlos Manzano, David Ruiz, J. Cabrera, César Díaz, P. Pérez, Narciso García","doi":"10.1109/QoMEX51781.2021.9465389","DOIUrl":"https://doi.org/10.1109/QoMEX51781.2021.9465389","url":null,"abstract":"This work-in-progress presents a dataset of 360degree videos, called EVENT-CLASS, with associated characteristics in the context of tele-education. The sequences (video and audio) have been captured considering several environments, lighting conditions, acquisition perspectives, and cameras, enriching the dataset. EVENT-CLASS will be helpful for numerous applications related to tele-education, including quality assessment tests, and with the aim of improving the immersive experience of remote users thanks to the detection of relevant events that happen in the class. In this sense, this paper presents preliminary results of using transfer learning for person detection in 360degree scenes, based on Detectron2, and provides insights on the influence of applying it to equirectangular projection and over the viewport. Ongoing works will allow to include more videos and ground-truth annotations to the dataset.","PeriodicalId":160378,"journal":{"name":"2021 13th International Conference on Quality of Multimedia Experience (QoMEX)","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"120901405","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Dominik Keller, Markus Vaalgamaa, E. Paajanen, Rakesh Rao Ramachandra Rao, Steve Goering, A. Raake
{"title":"Groovability: Using Groove as a Novel Measure for Audio QoE with the Example of Smartphones","authors":"Dominik Keller, Markus Vaalgamaa, E. Paajanen, Rakesh Rao Ramachandra Rao, Steve Goering, A. Raake","doi":"10.1109/QoMEX51781.2021.9465440","DOIUrl":"https://doi.org/10.1109/QoMEX51781.2021.9465440","url":null,"abstract":"Groove in music is a fundamental part of why humans entrain to it and enjoy it. Smartphones have become an important medium to listen to music. Especially when being with others, loudspeaker playback may be the method of choice. However, due to the physical limits of acoustics, for loudspeaker playback, smartphones are equipped with sub-optimal audio capabilities. Therefore, it is desirable to measure Quality of Experience (QoE) of music played on smartphones. While audio playback is often assessed in terms of sound quality, the aim of this work is to address QoE in terms of the meaning or effect that the audio has on the listener. A key component for the meaning of popular music is groove. Hence, in this paper, we study “groovability”, that is, the ability of a piece of audio technology to convey groove. To instantiate our novel audio QoE assessment method, we apply it to music played by 8 different smartphones. For this purpose, looped 4-bar loudness-aligned recordings from 24 music pieces of different intrinsic groove were played back on the different smartphones. Our test method uses a multi-stimulus comparison with synchronized playback capability. A total of 62 subjects evaluated groovability using two stimulus subsets. It was found that the proposed methodology is highly effective to distinguish between the groovability provided by the considered phones. In addition, a reduced-reference model is proposed to predict groovability, using a set of both acoustics-and music-groove related features. In our formal validation on unknown data, the model is shown to provide good prediction performance with a Pearson correlation of greater than 0.90.","PeriodicalId":160378,"journal":{"name":"2021 13th International Conference on Quality of Multimedia Experience (QoMEX)","volume":"46 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132333430","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Steven Schmidt, Saman Zadtootaghaj, Shijie Wang, S. Möller
{"title":"Towards the Influence of Audio Quality on Gaming Quality of Experience","authors":"Steven Schmidt, Saman Zadtootaghaj, Shijie Wang, S. Möller","doi":"10.1109/QoMEX51781.2021.9465450","DOIUrl":"https://doi.org/10.1109/QoMEX51781.2021.9465450","url":null,"abstract":"Humans are fascinated by video games for many years, which intrinsically immerse players in their virtual environments. Apart from the challenges offered by steadily new game concepts, aesthetically pleasing environments and characters, stories, and sound effects are highly important for the player experience. Recently, the new concept of cloud gaming, which offers users to play games executed on a cloud server remotely, is becoming increasingly popular. While cloud gaming offers many advantages, additional audio and video streaming poses many technical challenges. To ensure a satisfying Quality of Experience (QoE) of their customers, all stakeholders are interested in finding out which aspects of a gaming experience are of high importance and how resources can be optimally allocated. However, gaming QoE is a multidimensional construct including hedonic and pragmatic aspects and could be strongly influenced by interaction quality, video quality, and audio quality. While the impact of network and video encoding parameters of cloud gaming services was investigated in much detail in recent years, not many studies about the effect of audio quality on gaming experiences are available. Thus, in this paper, the impact of audio quality on gaming experience under different bitrate and packet loss conditions using two popular games is investigated. Therefore, a subjective experiment adhering to the ITU-T Rec. P.809 was conducted. The results show a significant impact of packet loss on audio quality and the overall gaming QoE. However, no significant effect of the bitrate, which was reduced to a minimum of 32 kbps, was revealed. Additionally, the influence of audio quality on gaming QoE was stronger for a game, which contained mainly diegetic effect sounds compared to a game containing few diegetic sounds and affect sound as background music.","PeriodicalId":160378,"journal":{"name":"2021 13th International Conference on Quality of Multimedia Experience (QoMEX)","volume":"157 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129204931","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Marie-Lena Eckert, Emanuël Habets, Olli S. Rummukainen
{"title":"Cognitive Load Estimation Based on Pupillometry in Virtual Reality with Uncontrolled Scene Lighting","authors":"Marie-Lena Eckert, Emanuël Habets, Olli S. Rummukainen","doi":"10.1109/QoMEX51781.2021.9465417","DOIUrl":"https://doi.org/10.1109/QoMEX51781.2021.9465417","url":null,"abstract":"Virtual reality (VR) technology enables and requires new ways of user experience testing in immersive environments. For various aspects of user experience, objective assessment of the cognitive load can be a useful parameter. With eye-tracking becoming a more widespread feature of current VR headsets, pupillometry is an appealing option to unobtrusively measure cognitive load during a VR experience. This paper shows that pupil size measured by an off-the-shelf VR headset with an integrated eye tracker positively correlates with the self-reported cognitive load during a standard n-back task adapted to a VR environment. To overcome the need for steady scene-lighting conditions, we present a method to correct for the light-induced pupil size changes, otherwise masking the cognitive load effects. Our results show that a commercially available VR headset with eye tracking can be used to measure the cognitive load in unpredictable lighting conditions without additional hardware.","PeriodicalId":160378,"journal":{"name":"2021 13th International Conference on Quality of Multimedia Experience (QoMEX)","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122618130","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}