Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing最新文献_第6页

Does Exposure to Diverse Perspectives Mitigate Biases in Crowdwork? An Explorative Study 接触不同视角是否能减轻众筹中的偏见?探索性研究

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2020-10-01 DOI: 10.1609/hcomp.v8i1.7474

Xiaoni Duan, Chien-Ju Ho, Ming Yin

引用次数: 14

Trainbot: A Conversational Interface to Train Crowd Workers for Delivering On-Demand Therapy Trainbot:一个会话界面，用于培训人群工作者提供按需治疗

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2020-10-01 DOI: 10.1609/hcomp.v8i1.7458

Tahir Abbas, Vassilis-Javed Khan, U. Gadiraju, P. Markopoulos

引用次数: 12

Impact of Algorithmic Decision Making on Human Behavior: Evidence from Ultimatum Bargaining 算法决策对人类行为的影响:来自最后通牒议价的证据

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2020-10-01 DOI: 10.1609/hcomp.v8i1.7462

Alexander Erlei, Franck Nekdem, Lukas Meub, Avishek Anand, U. Gadiraju

{"title":"Impact of Algorithmic Decision Making on Human Behavior: Evidence from Ultimatum Bargaining","authors":"Alexander Erlei, Franck Nekdem, Lukas Meub, Avishek Anand, U. Gadiraju","doi":"10.1609/hcomp.v8i1.7462","DOIUrl":"https://doi.org/10.1609/hcomp.v8i1.7462","url":null,"abstract":"Recent advances in machine learning have led to the widespread adoption of ML models for decision support systems. However, little is known about how the introduction of such systems affects the behavior of human stakeholders. This pertains both to the people using the system, as well as those who are affected by its decisions. To address this knowledge gap, we present a series of ultimatum bargaining game experiments comprising 1178 participants. We find that users are willing to use a black-box decision support system and thereby make better decisions. This translates into higher levels of cooperation and better market outcomes. However, because users under-weigh algorithmic advice, market outcomes remain far from optimal. Explanations increase the number of unique system inquiries, but users appear less willing to follow the system’s recommendation. People who negotiate with a user who has a decision support system, but cannot use one themselves, react to its introduction by demanding a better deal for themselves, thereby decreasing overall cooperation levels. This effect is largely driven by the percentage of participants who perceive the system’s availability as unfair. Interpretability mitigates perceptions of unfairness. Our findings highlight the potential for decision support systems to further human cooperation, but also the need for regulators to consider heterogeneous stakeholder reactions. In particular, higher levels of transparency might inadvertently hurt cooperation through changes in fairness perceptions.","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"82062429","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 24

Fast, Accurate, and Healthier: Interactive Blurring Helps Moderators Reduce Exposure to Harmful Content 快速，准确，更健康:交互式模糊帮助版主减少暴露于有害内容

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2020-10-01 DOI: 10.26153/TSW/10199

Anubrata Das, B. Dang, Matthew Lease

引用次数: 16

Privacy-Preserving Face Redaction Using Crowdsourcing 使用众包保护隐私的面部编辑

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2020-10-01 DOI: 10.25394/PGS.15052041.V1

Abdullah S Alshaibani, Sylvia T Carrell, Li-Hsin Tseng, Jungmin Shin, Alexander J. Quinn

{"title":"Privacy-Preserving Face Redaction Using Crowdsourcing","authors":"Abdullah S Alshaibani, Sylvia T Carrell, Li-Hsin Tseng, Jungmin Shin, Alexander J. Quinn","doi":"10.25394/PGS.15052041.V1","DOIUrl":"https://doi.org/10.25394/PGS.15052041.V1","url":null,"abstract":"Redaction of private information from images is the kind of tedious, yet context-independent, task for which crowdsourcing is especially well suited. Despite tremendous progress, machine learning is not keeping pace with the needs of sensitive applications in which inadvertent disclosure could have real-world consequences. Human workers can detect faces that machines cannot; however, an open call to crowds would entail disclosure. We present IntoFocus, a method for engaging crowd workers to redact faces from images without disclosing the facial identities of people depicted. The method works iteratively, starting with a heavily filtered form of the image, and gradually reducing the strength of the filter, with a different set of workers reviewing the image at each step. IntoFocus exploits the gap between the filter level at which a face becomes unidentifiable and the level at which it becomes undetectable. To calibrate the algorithm, we performed a perceptual study of detection and identification of faces in images filtered with the median filter. We present the system design, the results of the perception study, and the results of a summative evaluation of the system","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"87272690","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4

Batch Prioritization of Data Labeling Tasks for Training Classifiers 训练分类器数据标注任务的批量优先化

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2020-10-01 DOI: 10.1609/hcomp.v8i1.7476

Masanari Kimura, Kei Wakabayashi, Atsuyuki Morishima

{"title":"Batch Prioritization of Data Labeling Tasks for Training Classifiers","authors":"Masanari Kimura, Kei Wakabayashi, Atsuyuki Morishima","doi":"10.1609/hcomp.v8i1.7476","DOIUrl":"https://doi.org/10.1609/hcomp.v8i1.7476","url":null,"abstract":"In a data labeling process for building machine learning, the choice of labeling data instances is known to have a significant impact on the performance of classifiers. So far, the study of active learning has addressed the issue of how to choose the subset by prioritizing the data instances based on the state of the current classifier. However, the active learning approach has two drawbacks that (i) require a training loop to update the priorities of labeling tasks and (ii) require us to choose a specific active learner while we do not know the optimal classification model. In this paper, we propose a new framework of priority-aware labeling system that allows a parallel task assignment to crowd workers without assuming a particular classifier, which is based on novel methods called “batch prioritization” and “label expansion”. We conducted experiments with multiple datasets to examine the effectiveness of the approach and found that the proposed method improves the performance of the final classifiers more quickly than the active learning approach despite that the labeling tasks can be processed in a fully parallel manner.","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85802304","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

Analyzing Workers Performance in Online Mapping Tasks Across Web, Mobile, and Virtual Reality Platforms 分析跨Web、移动和虚拟现实平台的在线地图任务中的工作人员绩效

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2020-10-01 DOI: 10.1609/hcomp.v8i1.7472

Gerard van Alphen, S. Qiu, A. Bozzon, G. Houben

{"title":"Analyzing Workers Performance in Online Mapping Tasks Across Web, Mobile, and Virtual Reality Platforms","authors":"Gerard van Alphen, S. Qiu, A. Bozzon, G. Houben","doi":"10.1609/hcomp.v8i1.7472","DOIUrl":"https://doi.org/10.1609/hcomp.v8i1.7472","url":null,"abstract":"In online crowd mapping, crowd workers recruited through crowdsourcing marketplaces collect geographic data. Compared to traditional mapping methods, where workers physically explore the area, the benefit of using online crowd mapping is the potential to be cost-effective and time-efficient. Previous studies have focused on mapping urban objects using street-level imagery. However, they are specifically aimed at a single type of object, and only through web platforms. To the best of our knowledge, there is still a lack of understanding on how workers perform the mapping tasks through different platforms. Aiming to fill this knowledge gap, we investigate the worker performance across web, mobile, and virtual reality platforms by designing a multi-platform system for mapping urban objects using street-level imagery with novel methods for geo-location estimation. We design a preliminary study to show the feasibility of executing online mapping tasks on three platforms. The result demonstrates that the type of task and execution platform can affect the worker performance in terms of worker accuracy, execution time, user engagement, and cognitive load.","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"83681079","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Effective Operator Summaries Extraction 有效的操作员摘要提取

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2020-10-01 DOI: 10.1609/hcomp.v8i1.7468

Ido Nimni, David Sarne

{"title":"Effective Operator Summaries Extraction","authors":"Ido Nimni, David Sarne","doi":"10.1609/hcomp.v8i1.7468","DOIUrl":"https://doi.org/10.1609/hcomp.v8i1.7468","url":null,"abstract":"This paper proposes a heuristic algorithm for effectively summarizing the work of novice robot operators, e.g., ones recruited through crowdsourcing platforms, in search and rescue-like tasks. Such summaries can be used for many purposes, perhaps most notably for monitoring and evaluating an operator’s performance in settings where information gaps preclude automatic evaluation. The underlying idea of our method is dividing the task timeline into intervals, and extracting a subset of high-scoring and low-scoring segments within, using a heuristic scoring function. This results in a short effective summary of the operator’s work, based on which several other crowdworkers can evaluate her performance. The effectiveness of the proposed method was extensively evaluated and compared to a large set of alternative methods through a series of experiments in Amazon Mechanical Turk. The analysis of the results reveals that the proposed method outperforms all tested alternatives. Finally, we evaluate the performance one may achieve with the use of machine learning for predicting the operator’s performance in our domain. While this approach manages to reach a performance level similar to the one achieved with summaries, it requires an order-of-magnitude greater effort for training (measured in terms of crowdworkers time).","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85283154","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Verifying Extended Entity Relationship Diagrams with Open Tasks 用打开的任务验证扩展实体关系图

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2020-10-01 DOI: 10.1609/hcomp.v8i1.7471

M. Sabou, Klemens Käsznar, Markus Zlabinger, S. Biffl, D. Winkler

{"title":"Verifying Extended Entity Relationship Diagrams with Open Tasks","authors":"M. Sabou, Klemens Käsznar, Markus Zlabinger, S. Biffl, D. Winkler","doi":"10.1609/hcomp.v8i1.7471","DOIUrl":"https://doi.org/10.1609/hcomp.v8i1.7471","url":null,"abstract":"The verification of Extended Entity Relationship (EER) diagrams and other conceptual models that capture the design of information systems is crucial to ensure reliable systems. To scale up verification processes to larger groups of experts, Human Computation techniques were used focusing primarily on closed tasks, which constrain the number and variety of reported defects in favor of easy aggregation of derived judgements. To address this limitation of closed tasks, in this paper, we investigate EER verification (as instance of a broader family of model verification problems) with open tasks to extend the range of collected results. We also address the challenge of aggregating results of open tasks by proposing a follow-up HC task for defect validation. We evaluate our approach for HC-based EER Verification with open tasks in a set of experiments conducted with junior developers and show that (1) open tasks allow collecting a variety of insights that go beyond a manually built gold standard while still leading to good performance (F1=60%) and (2) HC-based validation can be reliably used for validating the results of open tasks (F1=84% compared to expert validation).","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"84131424","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Predicting Crowdworkers' Performance as Human-Sensors for Robot Navigation 预测众工作为机器人导航人类传感器的表现

Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing Pub Date : 2020-10-01 DOI: 10.1609/hcomp.v8i1.7467

Nir Machlev, David Sarne

{"title":"Predicting Crowdworkers' Performance as Human-Sensors for Robot Navigation","authors":"Nir Machlev, David Sarne","doi":"10.1609/hcomp.v8i1.7467","DOIUrl":"https://doi.org/10.1609/hcomp.v8i1.7467","url":null,"abstract":"This paper provides and evaluates a new paradigm for collaborative human-robot operation in search and rescue-like settings with information asymmetry. In particular, we focus on settings where the human, a crowdworker in our case, is used as a sensor, providing the route-planning module with essential environmental information. In such settings, the ability to predict the expected performance of the collaborating crowdworker in real-time is instrumental for maintaining a continuously high level of performance. Through an extensive set of experiments with crowdworkers recruited and interacted through Amazon Mechanical Turk, we show that effective online prediction is indeed possible, however only if distinguishing between two subpopulations of crowdworkers, termed ”operators” and ”sensors”, applying a different prediction model to each. Furthermore, we show that even the classification of crowdworkers to the two types can be carried out successfully in real-time, based merely on the first two minutes of collaboration. Finally, we demonstrate how the above abilities can be used for a more effective workers’ recruiting process, resulting in a substantially improved overall performance.","PeriodicalId":87339,"journal":{"name":"Proceedings of the ... AAAI Conference on Human Computation and Crowdsourcing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"78729818","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2