Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing最新文献_第8页

Fair Work: Crowd Work Minimum Wage with One Line of Code 公平工作:群众工作最低工资一行代码

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2019-10-28 DOI: 10.1609/hcomp.v7i1.5283

Mark E. Whiting, Grant Hugh, Michael S. Bernstein

引用次数: 73

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2019-10-28 DOI: 10.1609/hcomp.v7i1.5264

R. Qarout, Alessandro Checco, Gianluca Demartini, Kalina Bontcheva

{"title":"Platform-Related Factors in Repeatability and Reproducibility of Crowdsourcing Tasks","authors":"R. Qarout, Alessandro Checco, Gianluca Demartini, Kalina Bontcheva","doi":"10.1609/hcomp.v7i1.5264","DOIUrl":"https://doi.org/10.1609/hcomp.v7i1.5264","url":null,"abstract":"Crowdsourcing platforms provide a convenient and scalable way to collect human-generated labels on-demand. This data can be used to train Artificial Intelligence (AI) systems or to evaluate the effectiveness of algorithms. The datasets generated by means of crowdsourcing are, however, dependent on many factors that affect their quality. These include, among others, the population sample bias introduced by aspects like task reward, requester reputation, and other filters introduced by the task design.In this paper, we analyse platform-related factors and study how they affect dataset characteristics by running a longitudinal study where we compare the reliability of results collected with repeated experiments over time and across crowdsourcing platforms. Results show that, under certain conditions: 1) experiments replicated across different platforms result in significantly different data quality levels while 2) the quality of data from repeated experiments over time is stable within the same platform. We identify some key task design variables that cause such variations and propose an experimentally validated set of actions to counteract these effects thus achieving reliable and repeatable crowdsourced data collection experiments.","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2019-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"75588022","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 11

Gamification of Loop-Invariant Discovery from Code 从代码中发现循环不变性的游戏化

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2019-10-28 DOI: 10.1609/hcomp.v7i1.5277

Andrew T. Walter, Benjamin Boskin, Seth Cooper, P. Manolios

引用次数: 6

Testing Stylistic Interventions to Reduce Emotional Impact of Content Moderation Workers 测试文体干预以减少内容审核工作者的情绪影响

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2019-10-28 DOI: 10.1609/hcomp.v7i1.5270

S. Karunakaran, Rashmi Ramakrishan

引用次数: 12

Second Opinion: Supporting Last-Mile Person Identification with Crowdsourcing and Face Recognition 第二意见:用众包和人脸识别技术支持最后一英里的人识别

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2019-10-28 DOI: 10.1609/hcomp.v7i1.5272

V. Mohanty, Kareem Abdol-Hamid, C. Ebersohl, K. Luther

{"title":"Second Opinion: Supporting Last-Mile Person Identification with Crowdsourcing and Face Recognition","authors":"V. Mohanty, Kareem Abdol-Hamid, C. Ebersohl, K. Luther","doi":"10.1609/hcomp.v7i1.5272","DOIUrl":"https://doi.org/10.1609/hcomp.v7i1.5272","url":null,"abstract":"As AI-based face recognition technologies are increasingly adopted for high-stakes applications like locating suspected criminals, public concerns about the accuracy of these technologies have grown as well. These technologies often present a human expert with a shortlist of high-confidence candidate faces from which the expert must select correct match(es) while avoiding false positives, which we term the “last-mile problem.” We propose Second Opinion, a web-based software tool that employs a novel crowdsourcing workflow inspired by cognitive psychology, seed-gather-analyze, to assist experts in solving the last-mile problem. We evaluated Second Opinion with a mixed-methods lab study involving 10 experts and 300 crowd workers who collaborate to identify people in historical photos. We found that crowds can eliminate 75% of false positives from the highest-confidence candidates suggested by face recognition, and that experts were enthusiastic about using Second Opinion in their work. We also discuss broader implications for crowd–AI interaction and crowdsourced person identification.","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2019-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"80227197","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 9

Going against the (Appropriate) Flow: A Contextual Integrity Approach to Privacy Policy Analysis 违背(适当)流程:隐私政策分析的上下文完整性方法

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2019-10-28 DOI: 10.1609/hcomp.v7i1.5266

Yan Shvartzshnaider, Noah J. Apthorpe, N. Feamster, H. Nissenbaum

{"title":"Going against the (Appropriate) Flow: A Contextual Integrity Approach to Privacy Policy Analysis","authors":"Yan Shvartzshnaider, Noah J. Apthorpe, N. Feamster, H. Nissenbaum","doi":"10.1609/hcomp.v7i1.5266","DOIUrl":"https://doi.org/10.1609/hcomp.v7i1.5266","url":null,"abstract":"We present a method for analyzing privacy policies using the framework of contextual integrity (CI). This method allows for the systematized detection of issues with privacy policy statements that hinder readers’ ability to understand and evaluate company data collection practices. These issues include missing contextual details, vague language, and overwhelming possible interpretations of described information transfers. We demonstrate this method in two different settings. First, we compare versions of Facebook’s privacy policy from before and after the Cambridge Analytica scandal. Our analysis indicates that the updated policy still contains fundamental ambiguities that limit readers’ comprehension of Facebook’s data collection practices. Second, we successfully crowdsourced CI annotations of 48 excerpts of privacy policies from 17 companies with 141 crowdworkers. This indicates that regular users are able to reliably identify contextual information in privacy policy statements and that crowdsourcing can help scale our CI analysis method to a larger number of privacy policy statements.","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2019-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88570288","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 27

The Effects of Meaningful and Meaningless Explanations on Trust and Perceived System Accuracy in Intelligent Systems 智能系统中有意义和无意义解释对信任和感知系统准确性的影响

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2019-10-28 DOI: 10.1609/hcomp.v7i1.5284

Mahsan Nourani, Samia Kabir, Sina Mohseni, E. Ragan

{"title":"The Effects of Meaningful and Meaningless Explanations on Trust and Perceived System Accuracy in Intelligent Systems","authors":"Mahsan Nourani, Samia Kabir, Sina Mohseni, E. Ragan","doi":"10.1609/hcomp.v7i1.5284","DOIUrl":"https://doi.org/10.1609/hcomp.v7i1.5284","url":null,"abstract":"Machine learning and artificial intelligence algorithms can assist human decision making and analysis tasks. While such technology shows promise, willingness to use and rely on intelligent systems may depend on whether people can trust and understand them. To address this issue, researchers have explored the use of explainable interfaces that attempt to help explain why or how a system produced the output for a given input. However, the effects of meaningful and meaningless explanations (determined by their alignment with human logic) are not properly understood, especially with users who are non-experts in data science. Additionally, we wanted to explore how explanation inclusion and level of meaningfulness would affect the user’s perception of accuracy. We designed a controlled experiment using an image classification scenario with local explanations to evaluate and better understand these issues. Our results show that whether explanations are human-meaningful can significantly affect perception of a system’s accuracy independent of the actual accuracy observed from system usage. Participants significantly underestimated the system’s accuracy when it provided weak, less human-meaningful explanations. Therefore, for intelligent systems with explainable interfaces, this research demonstrates that users are less likely to accurately judge the accuracy of algorithms that do not operate based on human-understandable rationale.","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2019-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"82777977","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 62

Human Evaluation of Models Built for Interpretability 人类对可解释性模型的评价

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2019-10-28 DOI: 10.1609/hcomp.v7i1.5280

Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, S. Gershman, F. Doshi-Velez

引用次数: 95

A Hybrid Approach to Identifying Unknown Unknowns of Predictive Models 一种识别预测模型未知未知数的混合方法

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2019-10-28 DOI: 10.1609/hcomp.v7i1.5274

C. Vandenhof

引用次数: 8

Studying the "Wisdom of Crowds" at Scale 大规模研究“群体智慧”

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2019-10-28 DOI: 10.1609/hcomp.v7i1.5271

Camelia Simoiu, C. Sumanth, A. Mysore, Sharad Goel

{"title":"Studying the \"Wisdom of Crowds\" at Scale","authors":"Camelia Simoiu, C. Sumanth, A. Mysore, Sharad Goel","doi":"10.1609/hcomp.v7i1.5271","DOIUrl":"https://doi.org/10.1609/hcomp.v7i1.5271","url":null,"abstract":"In a variety of problem domains, it has been observed that the aggregate opinions of groups are often more accurate than those of the constituent individuals, a phenomenon that has been dubbed the “wisdom of the crowd”. However, due to the varying contexts, sample sizes, methodologies, and scope of previous studies, it has been difficult to gauge the extent to which conclusions generalize. To investigate this question, we carried out a large online experiment to systematically evaluate crowd performance on 1,000 questions across 50 topical domains. We further tested the effect of different types of social influence on crowd performance. For example, in one condition, participants could see the cumulative crowd answer before providing their own. In total, we collected more than 500,000 responses from nearly 2,000 participants. We have three main results. First, averaged across all questions, we find that the crowd indeed performs better than the average individual in the crowd—but we also find substantial heterogeneity in performance across questions. Second, we find that crowd performance is generally more consistent than that of individuals; as a result, the crowd does considerably better than individuals when performance is computed on a full set of questions within a domain. Finally, we find that social influence can, in some instances, lead to herding, decreasing crowd performance. Our findings illustrate some of the subtleties of the wisdom-of-crowds phenomenon, and provide insights for the design of social recommendation platforms.","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2019-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"86836418","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 10