Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia最新文献

{"title":"Impact of Tone-mapping Algorithms on Subjective and Objective Face Recognition in HDR Images","authors":"Pavel Korshunov, Marco V. Bernardo, A. Pinheiro, T. Ebrahimi","doi":"10.1145/2810188.2810195","DOIUrl":"https://doi.org/10.1145/2810188.2810195","url":null,"abstract":"Crowdsourcing is a popular tool for conducting subjective evaluations in uncontrolled environments and at low cost. In this paper, a crowdsourcing study is conducted to investigate the impact of High Dynamic Range (HDR) imaging on subjective face recognition accuracy. For that purpose, a dataset of HDR images of people depicted in high-contrast lighting conditions was created and their faces were manually cropped to construct a probe set of faces. Crowdsourcing-based face recognition was conducted for five differently tone-mapped versions of HDR faces and were compared to face recognition in a typical Low Dynamic Range alternative. A similar experiment was also conducted using three automatic face recognition algorithms. The comparative analysis results of face recognition by human subjects through crowdsourcing and machine vision face recognition show that HDR imaging affects the recognition results of human and computer vision approaches differently.","PeriodicalId":284531,"journal":{"name":"Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132332602","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Why Design Matters: Crowdsourcing of Complex Tasks","authors":"Bård Winther, M. Riegler, L. Calvet, C. Griwodz, P. Halvorsen","doi":"10.1145/2810188.2810190","DOIUrl":"https://doi.org/10.1145/2810188.2810190","url":null,"abstract":"In this paper, we show how the power of the crowd can be used to do complex tasks using human pose estimation as a use case. Crowdsourcing tasks are usually not very complex and easy to solve by workers that are not experienced in a certain topic. For more complex tasks, it is general recommended to use experienced workers or experts. However, we show that tasks can also be more complex for non-expert workers and that they produce data that is close to what experts would report. Therefore, a detailed description of the crowdsourcing campaign, the methods applied and the results will be given, and we discuss how the obtained data can be used.","PeriodicalId":284531,"journal":{"name":"Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134618563","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Do Scale-Design and Training Matter for Video QoE Assessments through Crowdsourcing?","authors":"B. Gardlo, S. Egger, T. Hossfeld","doi":"10.1145/2810188.2810193","DOIUrl":"https://doi.org/10.1145/2810188.2810193","url":null,"abstract":"Crowdsourcing (CS) has evolved into a mature assessment methodology for subjective experiments in diverse scientific fields and in particular for QoE assessment. However, the results acquired for absolute category rating (ACR) scales through CS are often not fully comparable to QoE assessments done in laboratory environments. A possible reason for such differences may be the scale usage heterogeneity problem caused by deviant scale usage of the crowd workers. In this paper, we study different implementations of (quality) rating scales (in terms of design and number of answer categories) in order to identify if certain scales can help to overcome scale usage problems in crowdsourcing. Additionally, training of subjects is well known to enhance result quality for laboratory ACR evaluations. Hence, we analyzed the appropriateness of training conditions to overcome scale usage problems across different samples in crowdsourcing. As major results, we found that filtering of user ratings and different scale designs are not sufficient to overcome scale usage heterogeneity, but training sessions despite their additional costs, enhance result quality in CS and properly counterfeit the identified scale usage heterogeneity problems.","PeriodicalId":284531,"journal":{"name":"Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129643984","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Eye Tracker in the Wild: Studying the delta between what is said and measured in a crowdsourcing experiment","authors":"Pierre R. Lebreton, I. Hupont, Toni Mäki, E. Skodras, Matthias Hirth","doi":"10.1145/2810188.2810192","DOIUrl":"https://doi.org/10.1145/2810188.2810192","url":null,"abstract":"Self-reported metrics collected in crowdsourcing experiments do not always match the actual user behaviour. Therefore in the laboratory studies the visual attention, the capability of humans to selectively process the visual information with which they are confronted, is traditionally measured by means of eye trackers. Visual attention has not been typically considered in crowdsourcing environments, mainly because of the requirements of specific hardware and challenging gaze calibration. This paper proposes the use of a non-intrusive eye tracking crowdsourcing framework, where the only technical requirements from the users' side are a webcam and a HTML5 compatible web browser, to study the differences between what a participant implicitly and explicitly does during a crowdsourcing experiment. To demonstrate the feasibility of this approach, an exemplary crowdsourcing campaign was launched to collect and compare both eye tracking data and self-reported metrics from the users. Participants performed a movie selection task, where they were asked about the main reasons motivating them to choose a particular movie. Results demonstrate the added value of monitoring gaze in crowdsourcing contexts: consciously or not, users behave differently than what they report through questionnaires.","PeriodicalId":284531,"journal":{"name":"Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia","volume":"41 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128684480","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Bridging the Utilitarian-Hedonic Divide in Crowdsourcing Applications","authors":"M. Melenhorst, J. Novak, I. Micheel, M. Larson, Martin Böckle","doi":"10.1145/2810188.2810191","DOIUrl":"https://doi.org/10.1145/2810188.2810191","url":null,"abstract":"This paper introduces a novel perspective on the gamification of crowdsourcing tasks by conceptualizing it as the introduction of hedonic quality into the solution of utilitarian tasks and into the design of corresponding systems. We demonstrate how such a conceptualization can enable crowdsourcing applications to involve new kinds of crowds in everyday contexts that cannot be reached with existing models. We illustrate its application with the design of TrendRack, a gamified crowdsourcing application in the domain of fashion. We then discuss the results from a first evaluation, suggesting successful engagement of fashion customers in everyday contexts.","PeriodicalId":284531,"journal":{"name":"Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126333662","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia","authors":"J. Redi, S. Rudinac","doi":"10.1145/2810188","DOIUrl":"https://doi.org/10.1145/2810188","url":null,"abstract":"Crowdsourcing has the potential to address key challenges in multimedia research. Multimedia evaluation, annotation, retrieval and creation can be obtained at a low time and monetary cost from the contribution of large crowds and by leveraging human computation. In fact, the applicative frontiers of this potential are yet to be discovered. And yet, challenges already arise as to how to cautiously exploit it. The crowd, as a users' (workers) community, is a complex and dynamic system highly sensitive to changes in the form and the parameterization of their activities. Issues concerning motivation, reliability, and engagement are being more and more often documented, and need to be addressed. \u0000 \u0000Since 2012, the International ACM Workshop on Crowdsourcing for Multimedia (CrowdMM) has welcomed new insights on the effective deployment of crowdsourcing towards boosting Multimedia research. On its fourth year, CrowdMM15 focuses on contributions addressing the key challenges that still hinder widespread adoption of crowdsourcing paradigms in the multimedia research community: remote monitoring of the user behavior, effective test design, controlling noise and quality in the results, designing incentive structures that do not breed cheating, and effective ways of keeping the user (the crowd!) in the loop to boost multimedia applications. \u0000 \u0000The call for papers attracted a good number of international submissions, two of which short papers. Of these, three were accepted as oral presentations and four as posters. All papers received at least three double blind reviews, and 3.5 reviews on average. \u0000 \u0000CrowdMM15 also proudly features the keynote talk of Prof. Shih-Fu Chang (Columbia University), addressing Crowdsourcing in video event detection, sentiment analysis and user intent modelling. Furthermore, for the second year this year CrowdMM proposes the Crowdkeynote: a crowd-sourced keynote, during which all members of the CrowdMM community give their view on the future and the Challenges that Crowdsourcing has still ahead. The slides of the Crowdkeynote can be found at https://goo.gl/Xlur2E.","PeriodicalId":284531,"journal":{"name":"Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia","volume":"70 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114695835","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"When the Crowd Challenges the Lab: Lessons Learnt from Subjective Studies on Image Aesthetic Appeal","authors":"J. Redi, E. Siahaan, Pavel Korshunov, Julian Habigt, T. Hossfeld","doi":"10.1145/2810188.2810194","DOIUrl":"https://doi.org/10.1145/2810188.2810194","url":null,"abstract":"Crowdsourcing gives researchers the opportunity to collect subjective data quickly, in the real-world, and from a very diverse pool of users. In a long-term study on image aesthetic appeal, we challenged the crowdsourced assessments with typical lab methodologies in order to identify and analyze the impact of crowdsourcing environment on the reliability of subjective data. We identified and conducted three types of crowdsourcing experiments that helped us perform an in-depth analysis of factors influencing reliability and reproducibility of results in uncontrolled crowdsourcing environments. We provide a generalized summary of lessons learnt for future research studies which will try to port lab-based evaluation methodologies into crowdsourcing, so that they can avoid the typical pitfalls in design and analysis of crowdsourcing experiments.","PeriodicalId":284531,"journal":{"name":"Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia","volume":"43 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132291443","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Generation of a Video Summary on a News Topic Based on SNS Responses to News Stories","authors":"Kosuke Kato, I. Ide, Daisuke Deguchi, H. Murase","doi":"10.1145/2810188.2810189","DOIUrl":"https://doi.org/10.1145/2810188.2810189","url":null,"abstract":"Archiving news videos is important since they accumulate valuable real-world information. When exploiting them, it is important to track the flow of news topics to understand them thoroughly. In order to do so, a method that structures the chronological semantic relations between news stories, namely the \"topic thread structure\" has been proposed in the past. However, simply viewing videos that compose this structure imposes a user to spend a long time watching detailed reports. On the other hand, Social Networking Services (SNS) have become very popular. SNS users often send and receive information in which they are interested while watching TV. Thus, we propose a method that automatically generates a video summary on a news topic from the general users' viewpoint based on responses of SNS users.","PeriodicalId":284531,"journal":{"name":"Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114707515","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Words and Pictures: Crowdsource Discovery beyond Image Semantics","authors":"Shih-Fu Chang","doi":"10.1145/2810188.2810197","DOIUrl":"https://doi.org/10.1145/2810188.2810197","url":null,"abstract":"Large annotated images from the Web and crowdsource, together with powerful machine learning tools, play a crucial role in rapid progress in semantic recognition of image data in recent years. However, as inferred in the saying \"A Picture is Worth More Than 1,000 Words,\" there is much richer information than just semantic labels associated with images from the Web resources and Crowdsource Fora. Such additional information covers the rich unexploited aspects, such as visual aesthetics, emotions, sentiments, user intention, and knowledge structure. Discovering such novel dimensions of image descriptions beyond semantics will have huge impact for exciting emerging applications such as personalized search and content recommendation. But it requires rigorous research in concept definition, task formulation, data crawling, and evaluation mechanisms. In this talk, I will address these issues by sharing our experiences [1-4] in discovering beyond-semantic visual descriptions related to visual sentiment, video upload intent classification, cultural influence on visual sentiment, and finally a wiki-style video event ontology.","PeriodicalId":284531,"journal":{"name":"Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia","volume":"146 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134491445","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}