Human computation (Fairfax, Va.)最新文献

{"title":"Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics","authors":"Lisa Posch, Arnim Bleier, Fabian Flöck, M. Strohmaier","doi":"10.15346/hc.v9i1.106","DOIUrl":"https://doi.org/10.15346/hc.v9i1.106","url":null,"abstract":"Since its emergence roughly a decade ago, micro-task crowdsourcing has been attracting a heterogeneous set of workers from all over the globe. This paper sets out to explore the characteristics of the international crowd workforce to date and offers a cross-national comparison of crowdworker populations from ten hand-selected countries. We provide an analysis and comparison of demographic characteristics and shed light on the significance of micro-task income for workers situated in different national contexts. With over 11,000 individual responses, this study is the first large-scale country-level analysis of the characteristics of workers on the platform Figure Eight (formerly CrowdFlower), one of the two platforms dominating the micro-task market. We find large differences between the characteristics of the crowd workforces of different countries, both regarding demography and regarding the importance of micro-task income for workers. Furthermore, we find that the composition of the workforce in the ten countries was largely stable across samples taken at different points in time.","PeriodicalId":92785,"journal":{"name":"Human computation (Fairfax, Va.)","volume":"50 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2018-12-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"89880547","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Human Computation vs. Machine Learning: an Experimental Comparison for Image Classification","authors":"Gloria Re Calegari, Gioele Nasi, I. Celino","doi":"10.15346/HC.V5I1.2","DOIUrl":"https://doi.org/10.15346/HC.V5I1.2","url":null,"abstract":"Image classification is a classical task heavily studied in computer vision and widely required in many concrete scientific and industrial scenarios. Is it better to rely on human eyes, thus asking people to classify pictures, or to train a machine learning system to automatically solve the task? The answer largely depends on the specific case and the required accuracy: humans may be more reliable - especially if they are domain experts - but automatic processing can be cheaper, even if less capable to demonstrate an \"intelligent\" behaviour.In this paper, we present an experimental comparison of different Human Computation and Machine Learning approaches to solve the same image classification task on a set of pictures used in light pollution research. We illustrate the adopted methods and the obtained results and we compare and contrast them in order to come up with a long term combined strategy to address the specific issue at scale: while it is hard to ensure a long-term engagement of users to exclusively rely on the Human Computation approach, the human classification is indispensable to overcome the \"cold start\" problem of automated data modelling.","PeriodicalId":92785,"journal":{"name":"Human computation (Fairfax, Va.)","volume":"440 1","pages":"13-30"},"PeriodicalIF":0.0,"publicationDate":"2018-07-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"76499615","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Chefs Know More than Just Recipes: Professional Vision in a Citizen Science Game","authors":"Marisa Ponti, Igor Stankovic, W. Barendregt, Bruno Kestemont, Lyn Bain","doi":"10.15346/HC.V5I1.1","DOIUrl":"https://doi.org/10.15346/HC.V5I1.1","url":null,"abstract":"Some citizen science projects use “games with a purpose” (GWAPs) to integrate what humans and computers, respectively, can do well. One of these projects is Foldit, which invites talented players to predict three-dimensional (3D) models of proteins from their amino acid composition. This study investigated players’ professional vision and interpret their use of recipes, small scripts of computer code that automate some protein folding processes, to carry out their strategies more easily when solving game puzzles. Specifically, this study examined when, how and why the players ran recipes when solving the puzzles, and what actions those recipes performed in the gameplay.Autoethnographic accounts of players at different levels of experience (beginner, intermediate, and expert) with playing the game were analyzed using a grounded theory approach. The analysis of what these players observed and did visualized the professional vision necessary to use recipes sensibly and effectively. The findings highlight three key abilities: (a) seeing beauty; (b) repairing errors made by recipes, and (c) monitoring a large quantity of information to perform actions effectively. This study indicates that players indeed have to develop a professional vision independent of what the game itself can highlight. This is related to the nature of the game where it seems impossible for the game developers to show the affordances, because they are unknown. Players must learn to see the affordances and develop a professional vision, which means that they have to learn these skills through gaming.","PeriodicalId":92785,"journal":{"name":"Human computation (Fairfax, Va.)","volume":"11 1","pages":"1-12"},"PeriodicalIF":0.0,"publicationDate":"2018-07-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"73174684","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Autocompletion interfaces make crowd workers slower, but their use promotes response diversity","authors":"Xipei Liu, James P. Bagrow","doi":"10.15346/hc.v6i1.3","DOIUrl":"https://doi.org/10.15346/hc.v6i1.3","url":null,"abstract":"Creative tasks such as ideation or question proposal are powerful applications of crowdsourcing, yet the quantity of workers available for addressing practical problems is often insufficient. To enable scalable crowdsourcing thus requires gaining all possible efficiency and information from available workers. One option for text-focused tasks is to allow assistive technology, such as an autocompletion user interface (AUI), to help workers input text responses. But support for the efficacy of AUIs is mixed. Here we designed and conducted a randomized experiment where workers were asked to provide short text responses to given questions. Our experimental goal was to determine if an AUI helps workers respond more quickly and with improved consistency by mitigating typos and misspellings. Surprisingly, we found that neither occurred: workers assigned to the AUI treatment were slower than those assigned to the non-AUI control and their responses were more diverse, not less, than those of the control. Both the lexical and semantic diversities of responses were higher, with the latter measured using word2vec. A crowdsourcer interested in worker speed may want to avoid using an AUI, but using an AUI to boost response diversity may be valuable to crowdsourcers interested in receiving as much novel information from workers as possible.","PeriodicalId":92785,"journal":{"name":"Human computation (Fairfax, Va.)","volume":"59 1","pages":"42-55"},"PeriodicalIF":0.0,"publicationDate":"2017-07-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"82114096","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Can you moderate an unreadable message? 'Blind' content moderation via human computation","authors":"Seth Frey, M. Bos, R. Sumner","doi":"10.15346/HC.V4I1.5","DOIUrl":"https://doi.org/10.15346/HC.V4I1.5","url":null,"abstract":"User-generated content (UGC) is fundamental to online social engagement, but eliciting and managing it come with many challenges. The special features of UGC moderation highlight many of the general challenges of human computation in general. They also emphasize how moderation and privacy interact: people have rights to both privacy and safety online, but it is difficult to provide one without violating the other: scanning a user's inbox for potentially malicious messages seems to imply access to all safe ones as well. Are privacy and safety opposed, or is it possible in some circumstance to guarantee the safety of anonymous content without access to that content. We demonstrate that such \"blind content moderation\" is possible in certain domains. Additionally, the methods we introduce offer safety guarantees, an expressive content space, and require no human moderation load: they are safe, expressive, and scalable Though it may seem preposterous to try moderating UGC without human- or machine-level access to it, human computation makes blind moderation possible. We establish this existence claim by defining two very different human computational methods, behavioral thresholding and reverse correlation . Each leverages the statistical and behavioral properties of so-called \"inappropriate content\" in different decision settings to moderate UGC without access to a message's meaning or intention. The first, behavioral thresholding, is shown to generalize the well-known ESP game.","PeriodicalId":92785,"journal":{"name":"Human computation (Fairfax, Va.)","volume":"30 1","pages":"78-106"},"PeriodicalIF":0.0,"publicationDate":"2017-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"84483491","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Is Dutch Auction Suitable for Decomposable Tasks in Competitive Crowdsourcing Markets?","authors":"S. Mridha, M. Bhattacharyya","doi":"10.15346/HC.V4I1.4","DOIUrl":"https://doi.org/10.15346/HC.V4I1.4","url":null,"abstract":"Competitive crowdsourcing is a popular strategic model for solving tasks in an efficient economical way. Competitive crowdsourcing platforms generally choose the winners for a single indivisible task following the conventional Dutch auction to minimize the cost. In this paper, we show that when the tasks are decomposable such mechanisms become biased for competitive crowdsourcing markets. As a solution to this, we suggest a variant of the Dutch auction (referred to as time-invariant Dutch auction) for solving decomposable tasks. Overall, some new insights about the mechanism design for competitive crowdsourcing markets are obtained.","PeriodicalId":92785,"journal":{"name":"Human computation (Fairfax, Va.)","volume":"3 1","pages":"71-77"},"PeriodicalIF":0.0,"publicationDate":"2017-05-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85313298","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Herding the Crowd: Using Automated Planning for Better Crowdsourced Planning","authors":"L. Manikonda, T. Chakraborti, Kartik Talamadupula, S. Kambhampati","doi":"10.15346/HC.V4I1.2","DOIUrl":"https://doi.org/10.15346/HC.V4I1.2","url":null,"abstract":"One subclass of human computation applications are those directed at tasks that involve planning (e.g. tour planning) and scheduling (e.g. conference scheduling). Interestingly, work on these systems shows that even primitive forms of automated oversight on the human contributors helps in significantly improving the effectiveness of the humans/crowd. In this paper, we argue that the automated oversight used in these systems can be viewed as a primitive automated planner, and that there are several opportunities for more sophisticated automated planning in effectively steering crowdsourced planning. Straightforward adaptation of current planning technology is however hampered by the mismatch between the capabilities of human workers and automated planners. We identify and partially address two important challenges that need to be overcome before such adaptation of planning technology can occur: (i) interpreting inputs of the human workers (and the requester) and (ii) steering or critiquing plans produced by the human workers, armed only with incomplete domain and preference models. To these ends, we describe the implementation of AI-MIX, a tour plan generation system that uses automated checks and alerts to improve the quality of plans created by human workers.","PeriodicalId":92785,"journal":{"name":"Human computation (Fairfax, Va.)","volume":"16 1","pages":"25-46"},"PeriodicalIF":0.0,"publicationDate":"2017-04-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"81652093","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Exploring the effects of non-monetary reimbursement for participants in HCI research","authors":"Sarah Wiseman, Anna L. Cox, Sandy J. J. Gould, Duncan P. Brumby","doi":"10.15346/HC.V4I1.1","DOIUrl":"https://doi.org/10.15346/HC.V4I1.1","url":null,"abstract":"When running experiments within the field of Human Computer Interaction (HCI) it is common practice to ask participants to come to a specified lab location, and reimburse them monetarily for their time and travel costs. This, however, is not the only means by which to encourage participation in scientific study. Citizen science projects, which encourage the public to become involved in scientific research, have had great success in getting people to act as sensors to collect data or to volunteer their idling computer or brain power to classify large data sets across a broad range of fields including biology, cosmology and physical and environmental science. This is often done without the expectation of payment. Additionally, data collection need not be done on behalf of an external researcher; the Quantified Self (QS) movement allows people to reflect on data they have collected about themselves. This too, then, is a form of non-reimbursed data collection. Here we investigate whether citizen HCI scientists and those interested in personal data produce reliable results compared to participants in more traditional lab-based studies. Through six studies, we explore how participation rates and data quality are affected by recruiting participants without monetary reimbursement: either by providing participants with data about themselves as reward (a QS approach), or by simply requesting help with no extrinsic reward (as in citizen science projects). We show that people are indeed willing to take part in online HCI research in the absence of extrinsic monetary reward, and that the data generated by participants who take part for selfless reasons, rather than for monetary reward, can be as high quality as data gathered in the lab and in addition may be of higher quality than data generated by participants given monetary reimbursement online. This suggests that large HCI experiments could be run online in the future, without having to incur the equally large reimbursement costs alongside the possibility of running experiments in environments outside of the lab.","PeriodicalId":92785,"journal":{"name":"Human computation (Fairfax, Va.)","volume":"43 1","pages":"1-24"},"PeriodicalIF":0.0,"publicationDate":"2017-04-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"73029756","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Topical Video Search: Analysing Video Concept Annotation through Crowdsourcing Games","authors":"R. Gligorov, M. Hildebrand, J. V. Ossenbruggen, Lora Aroyo, G. Schreiber","doi":"10.15346/HC.V4I1.3","DOIUrl":"https://doi.org/10.15346/HC.V4I1.3","url":null,"abstract":"Games with a purpose (GWAPs) are increasingly used in audio-visual collections as a mechanism for annotating videos through tagging. One such GWAP is Waisda? , a video labeling game where players tag streaming video and win points by reaching consensus on tags with other players. The open-ended and unconstrained manner of tagging in the fast-paced setting of the game has fundamental impact on the resulting tags. We find that Waisda? tags predominately describe visual objects and rarely refer to the topics of the videos. In this study we evaluate to what extent the tags entered by players can be regarded as topical descriptors of the video material.  Moreover, we characterize the quality of the user tags as topical descriptors with the aim to detect and filter out the bad ones. Our results show that after filtering,  game tags perform equally well compared to the manually crafted metadata when it comes to accessing the videos based on topic. An important consequence of this finding is that tagging games can provide a cost-effective alternative in situations when manual annotation by professionals is too costly.","PeriodicalId":92785,"journal":{"name":"Human computation (Fairfax, Va.)","volume":"20 1","pages":"47-70"},"PeriodicalIF":0.0,"publicationDate":"2017-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"75850979","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Agroecology: A Fertile Field for Human Computation","authors":"P. Hanappe, R. Dunlop, Annemie Maes, L. Steels, Nicolas Duval","doi":"10.15346/HC.V3I1.13","DOIUrl":"https://doi.org/10.15346/HC.V3I1.13","url":null,"abstract":"Agroecology -- the science of sustainable agriculture -- offers a new and positive perspective for the creation of sustainable food systems. We argue that, from the outset, it is important to involve citizens in this development to create the necessary, bottom-up support for change in agriculture and to re-establish the ties between food production and consumption. Human Computation and Citizen Science offer opportunities to include citizens in the planning, monitoring and evaluation of agro-ecosystems. In the P2P Food Lab project, we also seek new ways to engage them more creatively by setting up a shared online/offline platform in which they can learn, practice, innovate, and share observations on agroecological techniques. This paper gives a description and the underlying motivations of our ongoing work.","PeriodicalId":92785,"journal":{"name":"Human computation (Fairfax, Va.)","volume":"16 1","pages":"225-233"},"PeriodicalIF":0.0,"publicationDate":"2016-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85025480","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}