Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems最新文献_第10页

The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality 分歧反卷积:使机器学习性能指标符合现实

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems Pub Date : 2021-05-06 DOI: 10.1145/3411764.3445423

Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori B. Hashimoto, Michael S. Bernstein

{"title":"The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality","authors":"Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori B. Hashimoto, Michael S. Bernstein","doi":"10.1145/3411764.3445423","DOIUrl":"https://doi.org/10.1145/3411764.3445423","url":null,"abstract":"Machine learning classifiers for human-facing tasks such as comment toxicity and misinformation often score highly on metrics such as ROC AUC but are received poorly in practice. Why this gap? Today, metrics such as ROC AUC, precision, and recall are used to measure technical performance; however, human-computer interaction observes that evaluation of human-facing systems should account for people’s reactions to the system. In this paper, we introduce a transformation that more closely aligns machine learning classification metrics with the values and methods of user-facing performance measures. The disagreement deconvolution takes in any multi-annotator (e.g., crowdsourced) dataset, disentangles stable opinions from noise by estimating intra-annotator consistency, and compares each test set prediction to the individual stable opinions from each annotator. Applying the disagreement deconvolution to existing social computing datasets, we find that current metrics dramatically overstate the performance of many human-facing machine learning tasks: for example, performance on a comment toxicity task is corrected from .95 to .73 ROC AUC.","PeriodicalId":20451,"journal":{"name":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","volume":"112 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2021-05-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"89343242","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 89

LightTouch Gadgets: Extending Interactions on Capacitive Touchscreens by Converting Light Emission to Touch Inputs 轻触小工具:通过将光发射转换为触摸输入来扩展电容触摸屏上的交互

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems Pub Date : 2021-05-06 DOI: 10.1145/3411764.3445581

Kaori Ikematsu, Kunihiro Kato, Y. Kawahara

引用次数: 7

Effects of Communication Directionality and AI Agent Differences in Human-AI Interaction 人-人工智能交互中通信方向性和AI Agent差异的影响

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems Pub Date : 2021-05-06 DOI: 10.1145/3411764.3445256

Zahra Ashktorab, Casey Dugan, James M. Johnson, Qian Pan, Wei Zhang, Sadhana Kumaravel, Murray Campbell

{"title":"Effects of Communication Directionality and AI Agent Differences in Human-AI Interaction","authors":"Zahra Ashktorab, Casey Dugan, James M. Johnson, Qian Pan, Wei Zhang, Sadhana Kumaravel, Murray Campbell","doi":"10.1145/3411764.3445256","DOIUrl":"https://doi.org/10.1145/3411764.3445256","url":null,"abstract":"In Human-AI collaborative settings that are inherently interactive, direction of communication plays a role in how users perceive their AI partners. In an AI-driven cooperative game with partially observable information, players (be it the AI or the human player) require their actions to be interpreted accurately by the other player to yield a successful outcome. In this paper, we investigate social perceptions of AI agents with various directions of communication in a cooperative game setting. We measure subjective social perceptions (rapport, intelligence, and likeability) of participants towards their partners when participants believe they are playing with an AI or with a human and the nature of the communication (responsiveness and leading roles). We ran a large scale study on Mechanical Turk (n=199) of this collaborative game and find significant differences in gameplay outcome and social perception across different AI agents, different directions of communication and when the agent is perceived to be an AI/Human. We find that the bias against the AI that has been demonstrated in prior studies varies with the direction of the communication and with the AI agent.","PeriodicalId":20451,"journal":{"name":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","volume":"3 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2021-05-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"87821086","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 14

Seeing Beyond Expert Blind Spots: Online Learning Design for Scale and Quality 超越专家盲点:在线学习设计的规模和质量

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems Pub Date : 2021-05-06 DOI: 10.1145/3411764.3445045

Xu Wang, C. Rosé, K. Koedinger

引用次数: 11

Styling Words: A Simple and Natural Way to Increase Variability in Training Data Collection for Gesture Recognition 样式词:一种增加手势识别训练数据收集可变性的简单而自然的方法

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems Pub Date : 2021-05-06 DOI: 10.1145/3411764.3445457

Woojin Kang, Intaek Jung, Daeho Lee, Jin-Hyuk Hong

引用次数: 2

CoNotate: Suggesting Queries Based on Notes Promotes Knowledge Discovery CoNotate:基于笔记的查询建议促进了知识发现

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems Pub Date : 2021-05-06 DOI: 10.1145/3411764.3445618

Srishti Palani, Zijian Ding, Austin Nguyen, Andrew Chuang, S. Macneil, Steven W. Dow

引用次数: 14

Optimization-based User Support for Cinematographic Quadrotor Camera Target Framing 基于优化的电影四旋翼相机目标取景用户支持

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems Pub Date : 2021-05-06 DOI: 10.1145/3411764.3445568

Christoph Gebhardt, Otmar Hilliges

引用次数: 4

Generating the Presence of Remote Mourners: a Case Study of Funeral Webcasting in Japan 产生远程哀悼者的存在:日本葬礼网络直播的案例研究

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems Pub Date : 2021-05-06 DOI: 10.1145/3411764.3445617

Daisuke Uriu, Kenta Toshima, Minori Manabe, Takeru Yazaki, Takeshi Funatsu, Atsushi Izumihara, Zendai Kashino, Atsushi Hiyama, M. Inami

{"title":"Generating the Presence of Remote Mourners: a Case Study of Funeral Webcasting in Japan","authors":"Daisuke Uriu, Kenta Toshima, Minori Manabe, Takeru Yazaki, Takeshi Funatsu, Atsushi Izumihara, Zendai Kashino, Atsushi Hiyama, M. Inami","doi":"10.1145/3411764.3445617","DOIUrl":"https://doi.org/10.1145/3411764.3445617","url":null,"abstract":"Funerals are irreplaceable events, especially for bereaved family members and relatives. However, the COVID-19 pandemic has prevented many people worldwide from attending their loved ones’ funerals. The authors had the opportunity to assist one family faced with this predicament by webcasting and recording funeral rites held near Tokyo in June, 2020. Using our original 360-degree Telepresence system and smartphones running Zoom, we enabled the deceased’s elder siblings to remotely attend the funeral and did our utmost to make them feel present in the funeral hall. Despite the webcasting via Zoom contributing more to their remote attendances than our system, we discovered thoughtful findings which could be useful for designing remote funeral attendances. From the findings, we also discuss how HCI designers can contribute to this highly sensitive issue, weaving together knowledge from various domains including techno-spiritual practices, thanato-sensitive designs; and other religious and cultural aspects related to death rituals.","PeriodicalId":20451,"journal":{"name":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","volume":"61 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2021-05-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"89024958","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 15

XRgonomics: Facilitating the Creation of Ergonomic 3D Interfaces XRgonomics:促进人体工程学3D界面的创建

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems Pub Date : 2021-05-06 DOI: 10.1145/3411764.3445349

João Marcelo Evangelista Belo, A. Feit, Tiare M. Feuchtner, Kaj Grønbæk

引用次数: 32

Gesture Knitter: A Hand Gesture Design Tool for Head-Mounted Mixed Reality Applications 手势编织:一个用于头戴式混合现实应用的手势设计工具

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems Pub Date : 2021-05-06 DOI: 10.1145/3411764.3445766

George B. Mo, John J. Dudley, P. Kristensson

引用次数: 17