First Workshop on Insights from Negative Results in NLP最新文献_第3页

Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data 反事实增广的SNLI训练数据并不比未增广的数据产生更好的泛化

First Workshop on Insights from Negative Results in NLP Pub Date : 2020-10-09 DOI: 10.18653/v1/2020.insights-1.13

William Huang, Haokun Liu, Samuel R. Bowman

{"title":"Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data","authors":"William Huang, Haokun Liu, Samuel R. Bowman","doi":"10.18653/v1/2020.insights-1.13","DOIUrl":"https://doi.org/10.18653/v1/2020.insights-1.13","url":null,"abstract":"A growing body of work shows that models exploit annotation artifacts to achieve state-of-the-art performance on standard crowdsourced benchmarks—datasets collected from crowdworkers to create an evaluation task—while still failing on out-of-domain examples for the same task. Recent work has explored the use of counterfactually-augmented data—data built by minimally editing a set of seed examples to yield counterfactual labels—to augment training data associated with these benchmarks and build more robust classifiers that generalize better. However, Khashabi et al. (2020) find that this type of augmentation yields little benefit on reading comprehension tasks when controlling for dataset size and cost of collection. We build upon this work by using English natural language inference data to test model generalization and robustness and find that models trained on a counterfactually-augmented SNLI dataset do not generalize better than unaugmented datasets of similar size and that counterfactual augmentation can hurt performance, yielding models that are less robust to challenge examples. Counterfactual augmentation of natural language understanding data through standard crowdsourcing techniques does not appear to be an effective way of collecting training data and further innovation is required to make this general line of work viable.","PeriodicalId":441528,"journal":{"name":"First Workshop on Insights from Negative Results in NLP","volume":"60 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131011942","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 30

HINT3: Raising the bar for Intent Detection in the Wild 提示3:提高野外意图检测的门槛

First Workshop on Insights from Negative Results in NLP Pub Date : 2020-09-29 DOI: 10.18653/v1/2020.insights-1.16

Gaurav Arora, Chirag Jain, Manas Chaturvedi, Krupal Modi

引用次数: 16

On the Impact of Data Augmentation on Downstream Performance in Natural Language Processing 自然语言处理中数据增强对下游性能的影响

First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.12

Itsuki Okimura, Machel Reid, M. Kawano, Yutaka Matsuo

{"title":"On the Impact of Data Augmentation on Downstream Performance in Natural Language Processing","authors":"Itsuki Okimura, Machel Reid, M. Kawano, Yutaka Matsuo","doi":"10.18653/v1/2022.insights-1.12","DOIUrl":"https://doi.org/10.18653/v1/2022.insights-1.12","url":null,"abstract":"With in the broader scope of machine learning, data augmentation is a common strategy to improve generalization and robustness of machine learning models. While data augmentation has been widely used within computer vision, its use in the NLP has been been comparably rather limited. The reason for this is that within NLP, the impact of proposed data augmentation methods on performance has not been evaluated in a unified manner, and effective data augmentation methods are unclear. In this paper, we look to tackle this by evaluating the impact of 12 data augmentation methods on multiple datasets when finetuning pre-trained language models. We find minimal improvements when data sizes are constrained to a few thousand, with performance degradation when data size is increased. We also use various methods to quantify the strength of data augmentations, and find that these values, though weakly correlated with downstream performance, correlate negatively or positively depending on the task.Furthermore, we find a glaring lack of consistently performant data augmentations. This all alludes to the difficulty of data augmentations for NLP tasks and we are inclined to believe that static data augmentations are not broadly applicable given these properties.","PeriodicalId":441528,"journal":{"name":"First Workshop on Insights from Negative Results in NLP","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131181515","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 6

Label Errors in BANKING77 银行标签错误

First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.19

Cecilia Ying, Stephen Thomas

引用次数: 4

Replicability under Near-Perfect Conditions – A Case-Study from Automatic Summarization 近乎完美条件下的可复制性——一个来自自动摘要的案例研究

First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.23

Margot Mieskes

引用次数: 0

BPE beyond Word Boundary: How NOT to use Multi Word Expressions in Neural Machine Translation 超越词边界的BPE:如何在神经机器翻译中不使用多词表达式

First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.24

Dipesh Kumar, Avijit Thawani

引用次数: 1

How Much Do Modifications to Transformer Language Models Affect Their Ability to Learn Linguistic Knowledge? 对变形语言模型的修改对他们学习语言知识的能力有多大影响?

First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.6

Simeng Sun, Brian Dillon, Mohit Iyyer

{"title":"How Much Do Modifications to Transformer Language Models Affect Their Ability to Learn Linguistic Knowledge?","authors":"Simeng Sun, Brian Dillon, Mohit Iyyer","doi":"10.18653/v1/2022.insights-1.6","DOIUrl":"https://doi.org/10.18653/v1/2022.insights-1.6","url":null,"abstract":"Recent progress in large pretrained language models (LMs) has led to a growth of analyses examining what kinds of linguistic knowledge are encoded by these models. Due to computational constraints, existing analyses are mostly conducted on publicly-released LM checkpoints, which makes it difficult to study how various factors during training affect the models’ acquisition of linguistic knowledge. In this paper, we train a suite of small-scale Transformer LMs that differ from each other with respect to architectural decisions (e.g., self-attention configuration) or training objectives (e.g., multi-tasking, focal loss). We evaluate these LMs on BLiMP, a targeted evaluation benchmark of multiple English linguistic phenomena. Our experiments show that while none of these modifications yields significant improvements on aggregate, changes to the loss function result in promising improvements on several subcategories (e.g., detecting adjunct islands, correctly scoping negative polarity items). We hope our work offers useful insights for future research into designing Transformer LMs that more effectively learn linguistic knowledge.","PeriodicalId":441528,"journal":{"name":"First Workshop on Insights from Negative Results in NLP","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121223015","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Clustering Examples in Multi-Dataset Benchmarks with Item Response Theory 基于项目反应理论的多数据集基准聚类实例

First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.14

Pedro Rodriguez, Phu Mon Htut, John P. Lalor, João Sedoc

引用次数: 0

An Empirical study to understand the Compositional Prowess of Neural Dialog Models 理解神经对话模型合成能力的实证研究

First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.21

Vinayshekhar Bannihatti Kumar, Vaibhav Kumar, Mukul Bhutani, Alexander I. Rudnicky

引用次数: 1

Do Dependency Relations Help in the Task of Stance Detection? 依赖关系对姿态检测有帮助吗?

First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.2

A. T. Cignarella, C. Bosco, Paolo Rosso

引用次数: 1