Transactions of the Association for Computational Linguistics最新文献_第3页

OpenFact: Factuality Enhanced Open Knowledge Extraction OpenFact:事实增强的开放知识提取

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2023-06-01 DOI: 10.1162/tacl_a_00569

Linfeng Song, Ante Wang, Xiaoman Pan, Hongming Zhang, Dian Yu, Lifeng Jin, Haitao Mi, Jinsong Su, Yue Zhang, Dong Yu

{"title":"OpenFact: Factuality Enhanced Open Knowledge Extraction","authors":"Linfeng Song, Ante Wang, Xiaoman Pan, Hongming Zhang, Dian Yu, Lifeng Jin, Haitao Mi, Jinsong Su, Yue Zhang, Dong Yu","doi":"10.1162/tacl_a_00569","DOIUrl":"https://doi.org/10.1162/tacl_a_00569","url":null,"abstract":"We focus on the factuality property during the extraction of an OpenIE corpus named OpenFact, which contains more than 12 million high-quality knowledge triplets. We break down the factuality property into two important aspects—expressiveness and groundedness—and we propose a comprehensive framework to handle both aspects. To enhance expressiveness, we formulate each knowledge piece in OpenFact based on a semantic frame. We also design templates, extra constraints, and adopt human efforts so that most OpenFact triplets contain enough details. For groundedness, we require the main arguments of each triplet to contain linked Wikidata1 entities. A human evaluation suggests that the OpenFact triplets are much more accurate and contain denser information compared to OPIEC-Linked (Gashteovski et al., 2019), one recent high-quality OpenIE corpus grounded to Wikidata. Further experiments on knowledge base completion and knowledge base question answering show the effectiveness of OpenFact over OPIEC-Linked as supplementary knowledge to Wikidata as the major KG.","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"11 1","pages":"686-702"},"PeriodicalIF":10.9,"publicationDate":"2023-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42442182","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions 开放域问答系统在最小化问题上的对比一致性探讨

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2023-05-23 DOI: 10.1162/tacl_a_00591

Zhihan Zhang, W. Yu, Zheng Ning, Mingxuan Ju, Meng Jiang

{"title":"Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions","authors":"Zhihan Zhang, W. Yu, Zheng Ning, Mingxuan Ju, Meng Jiang","doi":"10.1162/tacl_a_00591","DOIUrl":"https://doi.org/10.1162/tacl_a_00591","url":null,"abstract":"Abstract Contrast consistency, the ability of a model to make consistently correct predictions in the presence of perturbations, is an essential aspect in NLP. While studied in tasks such as sentiment analysis and reading comprehension, it remains unexplored in open-domain question answering (OpenQA) due to the difficulty of collecting perturbed questions that satisfy factuality requirements. In this work, we collect minimally edited questions as challenging contrast sets to evaluate OpenQA models. Our collection approach combines both human annotation and large language model generation. We find that the widely used dense passage retriever (DPR) performs poorly on our contrast sets, despite fitting the training set well and performing competitively on standard test sets. To address this issue, we introduce a simple and effective query-side contrastive loss with the aid of data augmentation to improve DPR training. Our experiments on the contrast sets demonstrate that DPR’s contrast consistency is improved without sacrificing its accuracy on the standard test sets.1","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"11 1","pages":"1082-1096"},"PeriodicalIF":10.9,"publicationDate":"2023-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43293629","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2

Cross-functional Analysis of Generalization in Behavioral Learning 行为学习中泛化的跨职能分析

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2023-05-22 DOI: 10.1162/tacl_a_00590

Pedro Henrique Luz de Araujo, Benjamin Roth

{"title":"Cross-functional Analysis of Generalization in Behavioral Learning","authors":"Pedro Henrique Luz de Araujo, Benjamin Roth","doi":"10.1162/tacl_a_00590","DOIUrl":"https://doi.org/10.1162/tacl_a_00590","url":null,"abstract":"Abstract In behavioral testing, system functionalities underrepresented in the standard evaluation setting (with a held-out test set) are validated through controlled input-output pairs. Optimizing performance on the behavioral tests during training (behavioral learning) would improve coverage of phenomena not sufficiently represented in the i.i.d. data and could lead to seemingly more robust models. However, there is the risk that the model narrowly captures spurious correlations from the behavioral test suite, leading to overestimation and misrepresentation of model performance—one of the original pitfalls of traditional evaluation. In this work, we introduce BeLUGA, an analysis method for evaluating behavioral learning considering generalization across dimensions of different granularity levels. We optimize behavior-specific loss functions and evaluate models on several partitions of the behavioral test suite controlled to leave out specific phenomena. An aggregate score measures generalization to unseen functionalities (or overfitting). We use BeLUGA to examine three representative NLP tasks (sentiment analysis, paraphrase identification, and reading comprehension) and compare the impact of a diverse set of regularization and domain generalization methods on generalization performance.1","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"11 1","pages":"1066-1081"},"PeriodicalIF":10.9,"publicationDate":"2023-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41847615","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Bridging the Gap between Synthetic and Natural Questions via Sentence Decomposition for Semantic Parsing 基于语义分析的句子分解弥合合成问题与自然问题之间的鸿沟

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2023-05-01 DOI: 10.1162/tacl_a_00552

Yilin Niu, Fei Huang, W. Liu, Jianwei Cui, Bin Wang, Minlie Huang

{"title":"Bridging the Gap between Synthetic and Natural Questions via Sentence Decomposition for Semantic Parsing","authors":"Yilin Niu, Fei Huang, W. Liu, Jianwei Cui, Bin Wang, Minlie Huang","doi":"10.1162/tacl_a_00552","DOIUrl":"https://doi.org/10.1162/tacl_a_00552","url":null,"abstract":"Semantic parsing maps natural language questions into logical forms, which can be executed against a knowledge base for answers. In real-world applications, the performance of a parser is often limited by the lack of training data. To facilitate zero-shot learning, data synthesis has been widely studied to automatically generate paired questions and logical forms. However, data synthesis methods can hardly cover the diverse structures in natural languages, leading to a large gap in sentence structure between synthetic and natural questions. In this paper, we propose a decomposition-based method to unify the sentence structures of questions, which benefits the generalization to natural questions. Experiments demonstrate that our method significantly improves the semantic parser trained on synthetic data (+7.9% on KQA and +8.9% on ComplexWebQuestions in terms of exact match accuracy). Extensive analysis demonstrates that our method can better generalize to natural questions with novel text expressions compared with baselines. Besides semantic parsing, our idea potentially benefits other semantic understanding tasks by mitigating the distracting structure features. To illustrate this, we extend our method to the task of sentence embedding learning, and observe substantial improvements on sentence retrieval (+13.1% for Hit@1).","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"11 1","pages":"367-383"},"PeriodicalIF":10.9,"publicationDate":"2023-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42399980","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation 高质量非自回归文本生成的有向无环变压器预训练

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2023-04-24 DOI: 10.1162/tacl_a_00582

Fei Huang, Pei Ke, Minlie Huang

引用次数: 3

Expectations over Unspoken Alternatives Predict Pragmatic Inferences 对非言语选择的期望预测语用推理

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2023-04-07 DOI: 10.1162/tacl_a_00579

Jennifer Hu, R. Levy, Judith Degen, Sebastian Schuster

{"title":"Expectations over Unspoken Alternatives Predict Pragmatic Inferences","authors":"Jennifer Hu, R. Levy, Judith Degen, Sebastian Schuster","doi":"10.1162/tacl_a_00579","DOIUrl":"https://doi.org/10.1162/tacl_a_00579","url":null,"abstract":"Abstract Scalar inferences (SI) are a signature example of how humans interpret language based on unspoken alternatives. While empirical studies have demonstrated that human SI rates are highly variable—both within instances of a single scale, and across different scales—there have been few proposals that quantitatively explain both cross- and within-scale variation. Furthermore, while it is generally assumed that SIs arise through reasoning about unspoken alternatives, it remains debated whether humans reason about alternatives as linguistic forms, or at the level of concepts. Here, we test a shared mechanism explaining SI rates within and across scales: context-driven expectations about the unspoken alternatives. Using neural language models to approximate human predictive distributions, we find that SI rates are captured by the expectedness of the strong scalemate as an alternative. Crucially, however, expectedness robustly predicts cross-scale variation only under a meaning-based view of alternatives. Our results suggest that pragmatic inferences arise from context-driven expectations over alternatives, and these expectations operate at the level of concepts.1","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"11 1","pages":"885-901"},"PeriodicalIF":10.9,"publicationDate":"2023-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42869322","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design 众包隐含话语关系的设计选择:揭示任务设计引入的偏见

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2023-04-03 DOI: 10.1162/tacl_a_00586

Valentina Pyatkin, Frances Yung, Merel C. J. Scholman, Reut Tsarfaty, Ido Dagan, Vera Demberg

{"title":"Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design","authors":"Valentina Pyatkin, Frances Yung, Merel C. J. Scholman, Reut Tsarfaty, Ido Dagan, Vera Demberg","doi":"10.1162/tacl_a_00586","DOIUrl":"https://doi.org/10.1162/tacl_a_00586","url":null,"abstract":"Abstract Disagreement in natural language annotation has mostly been studied from a perspective of biases introduced by the annotators and the annotation frameworks. Here, we propose to analyze another source of bias—task design bias, which has a particularly strong impact on crowdsourced linguistic annotations where natural language is used to elicit the interpretation of lay annotators. For this purpose we look at implicit discourse relation annotation, a task that has repeatedly been shown to be difficult due to the relations’ ambiguity. We compare the annotations of 1,200 discourse relations obtained using two distinct annotation tasks and quantify the biases of both methods across four different domains. Both methods are natural language annotation tasks designed for crowdsourcing. We show that the task design can push annotators towards certain relations and that some discourse relation senses can be better elicited with one or the other annotation approach. We also conclude that this type of bias should be taken into account when training and testing models.","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"11 1","pages":"1014-1032"},"PeriodicalIF":10.9,"publicationDate":"2023-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47802644","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Compositional Zero-Shot Domain Transfer with Text-to-Text Models 文本到文本模型的组合零射击域转移

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2023-03-23 DOI: 10.1162/tacl_a_00585

Fangyu Liu, Qianchu Liu, Shruthi Bannur, Fernando Pérez-García, Naoto Usuyama, Shenmin Zhang, Tristan Naumann, A. Nori, Hoifung Poon, J. Alvarez-Valle, O. Oktay, Stephanie L. Hyland

{"title":"Compositional Zero-Shot Domain Transfer with Text-to-Text Models","authors":"Fangyu Liu, Qianchu Liu, Shruthi Bannur, Fernando Pérez-García, Naoto Usuyama, Shenmin Zhang, Tristan Naumann, A. Nori, Hoifung Poon, J. Alvarez-Valle, O. Oktay, Stephanie L. Hyland","doi":"10.1162/tacl_a_00585","DOIUrl":"https://doi.org/10.1162/tacl_a_00585","url":null,"abstract":"Abstract Label scarcity is a bottleneck for improving task performance in specialized domains. We propose a novel compositional transfer learning framework (DoT51) for zero-shot domain transfer. Without access to in-domain labels, DoT5 jointly learns domain knowledge (from masked language modelling of unlabelled in-domain free text) and task knowledge (from task training on more readily available general-domain data) in a multi-task manner. To improve the transferability of task training, we design a strategy named NLGU: We simultaneously train natural language generation (NLG) for in-domain label-to-data generation, which enables data augmentation for self-finetuning and natural language understanding (NLU) for label prediction. We evaluate DoT5 on the biomedical domain and the resource-lean subdomain of radiology, focusing on natural language inference, text summarization, and embedding learning. DoT5 demonstrates the effectiveness of compositional transfer learning through multi-task learning. In particular, DoT5 outperforms the current state-of-the-art in zero-shot transfer by over 7 absolute points in accuracy on RadNLI. We validate DoT5 with ablations and a case study demonstrating its ability to solve challenging NLI examples requiring in-domain expertise.","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"11 1","pages":"1097-1113"},"PeriodicalIF":10.9,"publicationDate":"2023-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43977430","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Evaluating Transformer Models and Human Behaviors on Chinese Character Naming 评价变形模型与汉字命名中的人类行为

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2023-03-22 DOI: 10.1162/tacl_a_00573

Xiaomeng Ma, Lingyu Gao

引用次数: 0

Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discovery 少即是多:通过因果发现减轻开放域对话响应生成模型的虚假关联

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2023-03-02 DOI: 10.1162/tacl_a_00561

Tao Feng, Lizhen Qu, Gholamreza Haffari

引用次数: 0