发布求助
文献互助 智能选刊 最新文献

First Workshop on Insights from Negative Results in NLP最新文献

筛选
英文 中文
Combining Extraction and Generation for Constructing Belief-Consequence Causal Links 结合提取与生成构建信念-结果因果关系
First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.22
M. Alexeeva, Allegra A. Beal, M. Surdeanu
{"title":"Combining Extraction and Generation for Constructing Belief-Consequence Causal Links","authors":"M. Alexeeva, Allegra A. Beal, M. Surdeanu","doi":"10.18653/v1/2022.insights-1.22","DOIUrl":"https://doi.org/10.18653/v1/2022.insights-1.22","url":null,"abstract":"In this paper, we introduce and justify a new task—causal link extraction based on beliefs—and do a qualitative analysis of the ability of a large language model—InstructGPT-3—to generate implicit consequences of beliefs. With the language model-generated consequences being promising, but not consistent, we propose directions of future work, including data collection, explicit consequence extraction using rule-based and language modeling-based approaches, and using explicitly stated consequences of beliefs to fine-tune or prompt the language model to produce outputs suitable for the task.","PeriodicalId":441528,"journal":{"name":"First Workshop on Insights from Negative Results in NLP","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124916509","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Evaluating the Practical Utility of Confidence-score based Techniques for Unsupervised Open-world Classification 评估基于置信度分数的无监督开放世界分类技术的实际效用
First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.3
Sopan Khosla, Rashmi Gangadharaiah
{"title":"Evaluating the Practical Utility of Confidence-score based Techniques for Unsupervised Open-world Classification","authors":"Sopan Khosla, Rashmi Gangadharaiah","doi":"10.18653/v1/2022.insights-1.3","DOIUrl":"https://doi.org/10.18653/v1/2022.insights-1.3","url":null,"abstract":"Open-world classification in dialog systems require models to detect open intents, while ensuring the quality of in-domain (ID) intent classification. In this work, we revisit methods that leverage distance-based statistics for unsupervised out-of-domain (OOD) detection. We show that despite their superior performance on threshold-independent metrics like AUROC on test-set, threshold values chosen based on the performance on a validation-set do not generalize well to the test-set, thus resulting in substantially lower performance on ID or OOD detection accuracy and F1-scores. Our analysis shows that this lack of generalizability can be successfully mitigated by setting aside a hold-out set from validation data for threshold selection (sometimes achieving relative gains as high as 100%). Extensive experiments on seven benchmark datasets show that this fix puts the performance of these methods at par with, or sometimes even better than, the current state-of-the-art OOD detection techniques.","PeriodicalId":441528,"journal":{"name":"First Workshop on Insights from Negative Results in NLP","volume":"215 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121277892","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 4
Pre-trained language models evaluating themselves - A comparative study 预训练语言模型自我评估——一项比较研究
First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.25
Philipp Koch, M. Aßenmacher, C. Heumann
{"title":"Pre-trained language models evaluating themselves - A comparative study","authors":"Philipp Koch, M. Aßenmacher, C. Heumann","doi":"10.18653/v1/2022.insights-1.25","DOIUrl":"https://doi.org/10.18653/v1/2022.insights-1.25","url":null,"abstract":"Evaluating generated text received new attention with the introduction of model-based metrics in recent years. These new metrics have a higher correlation with human judgments and seemingly overcome many issues of previous n-gram based metrics from the symbolic age. In this work, we examine the recently introduced metrics BERTScore, BLEURT, NUBIA, MoverScore, and Mark-Evaluate (Petersen). We investigate their sensitivity to different types of semantic deterioration (part of speech drop and negation), word order perturbations, word drop, and the common problem of repetition. No metric showed appropriate behaviour for negation, and further none of them was overall sensitive to the other issues mentioned above.","PeriodicalId":441528,"journal":{"name":"First Workshop on Insights from Negative Results in NLP","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129845894","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Challenges in including extra-linguistic context in pre-trained language models 在预训练的语言模型中包含语言外语境的挑战
First Workshop on Insights from Negative Results in NLP Pub Date : 1900-01-01 DOI: 10.18653/v1/2022.insights-1.18
Ionut-Teodor Sorodoc, Laura Aina, Gemma Boleda
{"title":"Challenges in including extra-linguistic context in pre-trained language models","authors":"Ionut-Teodor Sorodoc, Laura Aina, Gemma Boleda","doi":"10.18653/v1/2022.insights-1.18","DOIUrl":"https://doi.org/10.18653/v1/2022.insights-1.18","url":null,"abstract":"To successfully account for language, computational models need to take into account both the linguistic context (the content of the utterances) and the extra-linguistic context (for instance, the participants in a dialogue). We focus on a referential task that asks models to link entity mentions in a TV show to the corresponding characters, and design an architecture that attempts to account for both kinds of context. In particular, our architecture combines a previously proposed specialized module (an “entity library”) for character representation with transfer learning from a pre-trained language model. We find that, although the model does improve linguistic contextualization, it fails to successfully integrate extra-linguistic information about the participants in the dialogue. Our work shows that it is very challenging to incorporate extra-linguistic information into pre-trained language models.","PeriodicalId":441528,"journal":{"name":"First Workshop on Insights from Negative Results in NLP","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130031489","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
首页 上一页
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整，为了账户安全，请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们：info@booksci.cn Book学术提供免费学术资源搜索服务，方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号：481959085
Book学术官方微信