A Dataset of Crowdsourced Word Sequences: Collections and Answer Aggregation for Ground Truth Creation

Jiyi Li, Fumiyo Fukumoto
{"title":"A Dataset of Crowdsourced Word Sequences: Collections and Answer Aggregation for Ground Truth Creation","authors":"Jiyi Li, Fumiyo Fukumoto","doi":"10.18653/v1/D19-5904","DOIUrl":null,"url":null,"abstract":"The target outputs of many NLP tasks are word sequences. To collect the data for training and evaluating models, the crowd is a cheaper and easier to access than the oracle. To ensure the quality of the crowdsourced data, people can assign multiple workers to one question and then aggregate the multiple answers with diverse quality into a golden one. How to aggregate multiple crowdsourced word sequences with diverse quality is a curious and challenging problem. People need a dataset for addressing this problem. We thus create a dataset (CrowdWSA2019) which contains the translated sentences generated from multiple workers. We provide three approaches as the baselines on the task of extractive word sequence aggregation. Specially, one of them is an original one we propose which models the reliability of workers. We also discuss some issues on ground truth creation of word sequences which can be addressed based on this dataset.","PeriodicalId":129206,"journal":{"name":"Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP","volume":"42 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2019-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"11","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.18653/v1/D19-5904","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 11

Abstract

The target outputs of many NLP tasks are word sequences. To collect the data for training and evaluating models, the crowd is a cheaper and easier to access than the oracle. To ensure the quality of the crowdsourced data, people can assign multiple workers to one question and then aggregate the multiple answers with diverse quality into a golden one. How to aggregate multiple crowdsourced word sequences with diverse quality is a curious and challenging problem. People need a dataset for addressing this problem. We thus create a dataset (CrowdWSA2019) which contains the translated sentences generated from multiple workers. We provide three approaches as the baselines on the task of extractive word sequence aggregation. Specially, one of them is an original one we propose which models the reliability of workers. We also discuss some issues on ground truth creation of word sequences which can be addressed based on this dataset.
众包词序列的数据集:用于地面真相创建的集合和答案聚合
许多NLP任务的目标输出是词序列。要收集用于训练和评估模型的数据,crowd比oracle更便宜,也更容易访问。为了保证众包数据的质量,人们可以为一个问题分配多个工作者,然后将多个不同质量的答案聚合成一个黄金答案。如何对具有不同质量的多个众包词序列进行聚合是一个有趣而富有挑战性的问题。人们需要一个数据集来解决这个问题。因此,我们创建了一个数据集(CrowdWSA2019),其中包含由多个工人生成的翻译句子。我们提供了三种方法作为提取词序列聚合任务的基准。特别地,我们提出了一个新颖的模型,用来模拟工人的可靠性。我们还讨论了基于此数据集可以解决的词序列的地面真值创建问题。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信