Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.)最新文献_第2页

A Human-in-the-loop Workflow for Multi-Factorial Sensitivity Analysis of Algorithmic Rankers 算法排序器多因子敏感性分析的人在环工作流

Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.) Pub Date : 2023-06-18 DOI: 10.1145/3597465.3605221

Jun Yuan, Aritra Dasgupta

{"title":"A Human-in-the-loop Workflow for Multi-Factorial Sensitivity Analysis of Algorithmic Rankers","authors":"Jun Yuan, Aritra Dasgupta","doi":"10.1145/3597465.3605221","DOIUrl":"https://doi.org/10.1145/3597465.3605221","url":null,"abstract":"Algorithmic rankers are ubiquitously applied in automated decision systems such as hiring, admission, and loan-approval systems. Without appropriate explanations, decision-makers often cannot audit or trust algorithmic rankers' outcomes. In recent years, XAI (explainable AI) methods have focused on classification models, but there for algorithmic rankers, we are yet to develop state-of-the-art explanation methods. Moreover, explanations are also sensitive to changes in data and ranker properties, and decision-makers need transparent model diagnostics for calibrating the degree and impact of ranker sensitivity. To fulfill these needs, we take a dual approach of: i) designing explanations by transforming Shapley values for the simple form of a ranker based on linear weighted summation and ii) designing a human-in-the-loop sensitivity analysis workflow by simulating data whose attributes follow user-specified statistical distributions and correlations. We leverage a visualization interface to validate the transformed Shapley values and draw inferences from them by leveraging multi-factorial simulations, including data distributions, ranker parameters, and rank ranges.","PeriodicalId":92279,"journal":{"name":"Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.)","volume":"104 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"81807662","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Raven 乌鸦

Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.) Pub Date : 2023-06-18 DOI: 10.1145/3597465.3605219

Sadeem Alsudais, Avinash Kumar, Chen Li

{"title":"Raven","authors":"Sadeem Alsudais, Avinash Kumar, Chen Li","doi":"10.1145/3597465.3605219","DOIUrl":"https://doi.org/10.1145/3597465.3605219","url":null,"abstract":"Using GUI-based workflows for data analysis is an iterative process. During each iteration, an analyst makes changes to the workflow to improve it, generating a new version each time. The results produced by executing these versions are materialized to help users refer to them in the future. In many cases, a new version of the workflow, when submitted for execution, produces a result equivalent to that of a previous one. Identifying such equivalence can save computational resources and time by reusing the materialized result. One way to optimize the performance of executing a new version is to compare the current version with a previous one and test if they produce the same results using a workflow version equivalence verifier. As the number of versions grows, this testing can become a computational bottleneck. In this paper, we present Raven, an optimization framework to accelerate the execution of a new version request by detecting and reusing the results of previous equivalent versions with the help of a version equivalence verifier. Raven ranks and prunes the set of prior versions to quickly identify those that may produce an equivalent result to the version execution request. Additionally, when the verifier performs computation to verify the equivalence of a version pair, there may be a significant overlap with previously tested version pairs. Raven identifies and avoids such repeated computations by extending the verifier to reuse previous knowledge of equivalence tests. We evaluated the effectiveness of Raven compared to baselines on real workflows and datasets.","PeriodicalId":92279,"journal":{"name":"Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.)","volume":"32 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"81405681","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Data Makes Better Data Scientists 数据造就更好的数据科学家

Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.) Pub Date : 2023-06-18 DOI: 10.1145/3597465.3605228

Jinjin Zhao, A. Gal, Sanjay Krishnan

引用次数: 0

Proceedings of the Workshop on Human-In-the-Loop Data Analytics 人在循环数据分析研讨会论文集

Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.) Pub Date : 2023-01-01 DOI: 10.1145/3597465

引用次数: 0

Designing the evaluation of operator-enabled interactive data exploration in VALIDE 设计在validate中启用操作员的交互式数据探索的评估

Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.) Pub Date : 2022-06-12 DOI: 10.1145/3546930.3547509

Yogendra Patil, S. Amer-Yahia, S. Subramanian

引用次数: 1

Towards causal physical error discovery in video analytics systems 视频分析系统中的因果物理错误发现

Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.) Pub Date : 2022-06-12 DOI: 10.1145/3546930.3547495

Ted Shaowang, Jinjin Zhao, Stavros Sintos, S. Krishnan

引用次数: 0

Flexible and scalable annotation tool to develop scene understanding datasets 灵活和可扩展的注释工具，以开发场景理解数据集

Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.) Pub Date : 2022-06-12 DOI: 10.1145/3546930.3547499

Md. Fazle Elahi Khan, Renran Tian, Xiao Luo

{"title":"Flexible and scalable annotation tool to develop scene understanding datasets","authors":"Md. Fazle Elahi Khan, Renran Tian, Xiao Luo","doi":"10.1145/3546930.3547499","DOIUrl":"https://doi.org/10.1145/3546930.3547499","url":null,"abstract":"Recent progress in data-driven vision and language-based tasks demands developing training datasets enriched with multiple modalities representing human intelligence. The link between text and image data is one of the crucial modalities for developing AI models. The development process of such datasets in the video domain requires much effort from researchers and annotators (experts and non-experts). Researchers re-design annotation tools to extract knowledge from annotators to answer new research questions. The whole process repeats for each new question which is time-consuming. However, since the last decade, there has been little change in how the researchers and annotators interact with the annotation process. We revisit the annotation workflow and propose a concept of an adaptable and scalable annotation tool. The concept emphasizes its users' interactivity to make annotation process design seamless and efficient. Researchers can conveniently add newer modalities to or augment the extant datasets using the tool. The annotators can efficiently link free-form text to image objects. For conducting human-subject experiments on any scale, the tool supports the data collection for attaining group ground truth. We have conducted a case study using a prototype tool between two groups with the participation of 74 non-expert people. We find that the interactive linking of free-form text to image objects feels intuitive and evokes a thought process resulting in a high-quality annotation. The new design shows ≈ 35% improvement in the data annotation quality. On UX evaluation, we receive above-average positive feedback from 25 people regarding convenience, UI assistance, usability, and satisfaction.","PeriodicalId":92279,"journal":{"name":"Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.)","volume":"83 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2022-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"84267918","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Enabling useful provenance in scripting languages with a human-in-the-loop 在脚本语言中使用人在循环中启用有用的来源

Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.) Pub Date : 2022-06-12 DOI: 10.1145/3546930.3547494

Yuze Lou, Michael J. Cafarella

{"title":"Enabling useful provenance in scripting languages with a human-in-the-loop","authors":"Yuze Lou, Michael J. Cafarella","doi":"10.1145/3546930.3547494","DOIUrl":"https://doi.org/10.1145/3546930.3547494","url":null,"abstract":"Most data scientists must build substantial data pipelines using scripting languages like Python and R. These pipelines are hard to get correct due to the large volume of data they process (thus the long execution time), and the fact that they are tested mainly by inspection of output data quality. It is therefore crucial for developers to reason about data through each step in the pipeline, starting from the raw input; this information is akin to data provenance in a relational setting. Past efforts for capturing data provenance for scripting languages have required substantial manual modifications to the scripts, or else yield information that is too inflexible for many debugging tasks. We instead propose a \"human-in-the-loop\" provenance generation model with three key improvements: (1) allowing humans to express the desired provenance through a provenance schema, (2) enabling one-time execution capture of scripts to produce traces that are later combined with different provenance schemata to yield useful provenance for different tasks, (3) providing a modular rule-based recommendation component to help design provenance schemata through a user interaction interface. We describe the concepts, the user experience with our system, explain the system components, and present preliminary experiment results.","PeriodicalId":92279,"journal":{"name":"Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.)","volume":"32 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2022-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"90435972","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Opportunities and risks for engaging research participants with self-logged menstrual health data 让研究参与者参与自我记录的月经健康数据的机会和风险

Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.) Pub Date : 2022-06-12 DOI: 10.1145/3546930.3547501

Samantha Robertson, K. Harley, Niloufar Salehi

{"title":"Opportunities and risks for engaging research participants with self-logged menstrual health data","authors":"Samantha Robertson, K. Harley, Niloufar Salehi","doi":"10.1145/3546930.3547501","DOIUrl":"https://doi.org/10.1145/3546930.3547501","url":null,"abstract":"Many people use health tracking apps to keep track of their menstrual cycles, often in the hopes of better understanding their own health, and being able to identify when something might be wrong. However, it can be very difficult to interpret this data alone. Meanwhile, it is becoming increasingly common for researchers to use data from these apps to learn more about menstrual health. In this work we ask, how could more participatory approaches to conducting menstrual health research benefit both participants and researchers? We identify key challenges and risks of this kind of engagement, and propose four design guidelines for human-in-the-loop data analysis tools that engage participants with large-scale, quantitative menstrual health research: surface and elicit feedback on the data cleaning and analysis procedure; convey information relative to other users and clinical guidance; structure engagement to ensure valid analyses; and support social engagement and learning. For each of these, we highlight key open research questions relevant to the HILDA and visualization research communities. We plan to for evaluate and iterate on these guidelines through design workshops with users, researchers, and healthcare providers.","PeriodicalId":92279,"journal":{"name":"Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.)","volume":"104 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2022-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"76678035","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

HumanAL HumanAL

Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. Workshop on Human-In-the-Loop Data Analytics (2nd : 2017 : Chicago, Ill.) Pub Date : 2022-06-12 DOI: 10.1145/3546930.3547496

Roee Shraga

引用次数: 3