Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances最新文献

筛选
英文 中文
Data Cleaning Tools for Token Classification Tasks 令牌分类任务的数据清理工具
K. Muthuraman, Frederick Reiss, Hong Xu, Bryan Cutler, Zachary Eichenberger
{"title":"Data Cleaning Tools for Token Classification Tasks","authors":"K. Muthuraman, Frederick Reiss, Hong Xu, Bryan Cutler, Zachary Eichenberger","doi":"10.18653/V1/2021.DASH-1.10","DOIUrl":"https://doi.org/10.18653/V1/2021.DASH-1.10","url":null,"abstract":"Human-in-the-loop systems for cleaning NLP training data rely on automated sieves to isolate potentially-incorrect labels for manual review. We have developed a novel technique for flagging potentially-incorrect labels with high sensitivity in named entity recognition corpora. We incorporated our sieve into an end-to-end system for cleaning NLP corpora, implemented as a modular collection of Jupyter notebooks built on extensions to the Pandas DataFrame library. We used this system to identify incorrect labels in the CoNLL-2003 corpus for English-language named entity recognition (NER), one of the most influential corpora for NER model research. Unlike previous work that only looked at a subset of the corpus’s validation fold, our automated sieve enabled us to examine the entire corpus in depth. Across the entire CoNLL-2003 corpus, we identified over 1300 incorrect labels (out of 35089 in the corpus). We have published our corrections, along with the code we used in our experiments. We are developing a repeatable version of the process we used on the CoNLL-2003 corpus as an open-source library.","PeriodicalId":248359,"journal":{"name":"Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125561859","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 5
Bridging Multi-disciplinary Collaboration Challenges in ML Development via Domain Knowledge Elicitation 通过领域知识启发来弥合机器学习开发中的多学科协作挑战
Soya Park
{"title":"Bridging Multi-disciplinary Collaboration Challenges in ML Development via Domain Knowledge Elicitation","authors":"Soya Park","doi":"10.18653/V1/2021.DASH-1.7","DOIUrl":"https://doi.org/10.18653/V1/2021.DASH-1.7","url":null,"abstract":"Building a machine learning model in a sophisticated domain is a time-consuming process, partially due to the steep learning curve of domain knowledge for data scientists. We introduce Ziva, an interface for supporting domain knowledge from domain experts to data scientists in two ways: (1) a concept creation interface where domain experts extract important concept of the domain and (2) five kinds of justification elicitation interfaces that solicit elicitation how the domain concept are expressed in data instances.","PeriodicalId":248359,"journal":{"name":"Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances","volume":"46 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132885007","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Human-In-The-LoopEntity Linking for Low Resource Domains 低资源域的Human-In-The-LoopEntity链接
Jan-Christoph Klie, Richard Eckart de Castilho, Iryna Gurevych
{"title":"Human-In-The-LoopEntity Linking for Low Resource Domains","authors":"Jan-Christoph Klie, Richard Eckart de Castilho, Iryna Gurevych","doi":"10.18653/V1/2021.DASH-1.6","DOIUrl":"https://doi.org/10.18653/V1/2021.DASH-1.6","url":null,"abstract":"Entity linking (EL) is concerned with disambiguating entity mentions in a text against knowledge bases (KB). To quickly annotate texts with EL even in low-resource domains and noisy text, we present a novel Human-In-The-Loop EL approach. We show that it greatly outperforms a strong baseline in simulation. In a user study, annotation time is reduced by 35 % compared to annotating without interactive support; users report that they strongly prefer our system over ones without. An open-source and ready-to-use implementation based on the text annotation platform is made available.","PeriodicalId":248359,"journal":{"name":"Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances","volume":"2674 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123176268","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Active learning and negative evidence for language identification 主动学习和语言识别的消极证据
Thomas Lippincott, Benjamin Van Durme
{"title":"Active learning and negative evidence for language identification","authors":"Thomas Lippincott, Benjamin Van Durme","doi":"10.18653/V1/2021.DASH-1.8","DOIUrl":"https://doi.org/10.18653/V1/2021.DASH-1.8","url":null,"abstract":"Language identification (LID), the task of determining the natural language of a given text, is an essential first step in most NLP pipelines. While generally a solved problem for documents of sufficient length and languages with ample training data, the proliferation of microblogs and other social media has made it increasingly common to encounter use-cases that *don’t* satisfy these conditions. In these situations, the fundamental difficulty is the lack of, and cost of gathering, labeled data: unlike some annotation tasks, no single “expert” can quickly and reliably identify more than a handful of languages. This leads to a natural question: can we gain useful information when annotators are only able to *rule out* languages for a given document, rather than supply a positive label? What are the optimal choices for gathering and representing such *negative evidence* as a model is trained? In this paper, we demonstrate that using negative evidence can improve the performance of a simple neural LID model. This improvement is sensitive to policies of how the evidence is represented in the loss function, and for deciding which annotators to employ given the instance and model state. We consider simple policies and report experimental results that indicate the optimal choices for this task. We conclude with a discussion of future work to determine if and how the results generalize to other classification tasks.","PeriodicalId":248359,"journal":{"name":"Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances","volume":"97 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122580196","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 2
It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models 对于大规模NLU模型,最好是验证:有人类参与的半监督学习
Verena Weber, Enrico Piovano, Melanie Bradford
{"title":"It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models","authors":"Verena Weber, Enrico Piovano, Melanie Bradford","doi":"10.18653/V1/2021.DASH-1.2","DOIUrl":"https://doi.org/10.18653/V1/2021.DASH-1.2","url":null,"abstract":"When a NLU model is updated, new utter- ances must be annotated to be included for training. However, manual annotation is very costly. We evaluate a semi-supervised learning workflow with a human in the loop in a produc- tion environment. The previous NLU model predicts the annotation of the new utterances, a human then reviews the predicted annotation. Only when the NLU prediction is assessed as incorrect the utterance is sent for human anno- tation. Experimental results show that the pro- posed workflow boosts the performance of the NLU model while significantly reducing the annotation volume. Specifically, in our setup, we see improvements of up to 14.16% for a recall-based metric and up to 9.57% for a F1- score based metric, while reducing the annota- tion volume by 97% and overall cost by 60% for each iteration.","PeriodicalId":248359,"journal":{"name":"Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121103680","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 3
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书