Transactions of the Association for Computational Linguistics最新文献_第7页

An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding 语言理解的端到端对比自监督学习框架

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2022-11-01 DOI: 10.1162/tacl_a_00521

Hongchao Fang, P. Xie

{"title":"An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding","authors":"Hongchao Fang, P. Xie","doi":"10.1162/tacl_a_00521","DOIUrl":"https://doi.org/10.1162/tacl_a_00521","url":null,"abstract":"Abstract Self-supervised learning (SSL) methods such as Word2vec, BERT, and GPT have shown great effectiveness in language understanding. Contrastive learning, as a recent SSL approach, has attracted increasing attention in NLP. Contrastive learning learns data representations by predicting whether two augmented data instances are generated from the same original data example. Previous contrastive learning methods perform data augmentation and contrastive learning separately. As a result, the augmented data may not be optimal for contrastive learning. To address this problem, we propose a four-level optimization framework that performs data augmentation and contrastive learning end-to-end, to enable the augmented data to be tailored to the contrastive learning task. This framework consists of four learning stages, including training machine translation models for sentence augmentation, pretraining a text encoder using contrastive learning, finetuning a text classification model, and updating weights of translation data by minimizing the validation loss of the classification model, which are performed in a unified way. Experiments on datasets in the GLUE benchmark (Wang et al., 2018a) and on datasets used in Gururangan et al. (2020) demonstrate the effectiveness of our method.","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"10 1","pages":"1324-1340"},"PeriodicalIF":10.9,"publicationDate":"2022-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47193269","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Helpful Neighbors: Leveraging Neighbors in Geographic Feature Pronunciation 有帮助的邻居:利用邻居的地理特征发音

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2022-10-18 DOI: 10.1162/tacl_a_00535

Llion Jones, R. Sproat, Haruko Ishikawa, Alexander Gutkin

引用次数: 1

Improving Low-Resource Cross-lingual Parsing with Expected Statistic Regularization 利用期望统计正则化改进低资源跨语言解析

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2022-10-17 DOI: 10.1162/tacl_a_00537

Thomas Effland, Michael Collins

引用次数: 1

Transparency Helps Reveal When Language Models Learn Meaning 透明度有助于揭示语言模型何时学习意义

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2022-10-14 DOI: 10.1162/tacl_a_00565

Zhaofeng Wu, Will Merrill, Hao Peng, Iz Beltagy, Noah A. Smith

引用次数: 2

Explainable Abuse Detection as Intent Classification and Slot Filling 可解释的滥用检测:意图分类和槽填充

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2022-10-06 DOI: 10.1162/tacl_a_00527

Agostina Calabrese, Björn Ross, Mirella Lapata

引用次数: 4

Domain-Specific Word Embeddings with Structure Prediction 具有结构预测的领域特定词嵌入

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2022-10-06 DOI: 10.1162/tacl_a_00538

Stephanie Brandl, D. Lassner, A. Baillot, S. Nakajima

{"title":"Domain-Specific Word Embeddings with Structure Prediction","authors":"Stephanie Brandl, D. Lassner, A. Baillot, S. Nakajima","doi":"10.1162/tacl_a_00538","DOIUrl":"https://doi.org/10.1162/tacl_a_00538","url":null,"abstract":"Complementary to finding good general word embeddings, an important question for representation learning is to find dynamic word embeddings, for example, across time or domain. Current methods do not offer a way to use or predict information on structure between sub-corpora, time or domain and dynamic embeddings can only be compared after post-alignment. We propose novel word embedding methods that provide general word representations for the whole corpus, domain- specific representations for each sub-corpus, sub-corpus structure, and embedding alignment simultaneously. We present an empirical evaluation on New York Times articles and two English Wikipedia datasets with articles on science and philosophy. Our method, called Word2Vec with Structure Prediction (W2VPred), provides better performance than baselines in terms of the general analogy tests, domain-specific analogy tests, and multiple specific word embedding evaluations as well as structure prediction performance when no structure is given a priori. As a use case in the field of Digital Humanities we demonstrate how to raise novel research questions for high literature from the German Text Archive.","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"11 1","pages":"320-335"},"PeriodicalIF":10.9,"publicationDate":"2022-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43780471","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering 面向开放领域问答的检索增强生成(RAG)模型的领域适应性改进

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2022-10-06 DOI: 10.1162/tacl_a_00530

Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, R. Rana, Suranga Nanayakkara

{"title":"Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering","authors":"Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, R. Rana, Suranga Nanayakkara","doi":"10.1162/tacl_a_00530","DOIUrl":"https://doi.org/10.1162/tacl_a_00530","url":null,"abstract":"Retrieval Augment Generation (RAG) is a recent advancement in Open-Domain Question Answering (ODQA). RAG has only been trained and explored with a Wikipedia-based external knowledge base and is not optimized for use in other specialized domains such as healthcare and news. In this paper, we evaluate the impact of joint training of the retriever and generator components of RAG for the task of domain adaptation in ODQA. We propose RAG-end2end, an extension to RAG that can adapt to a domain-specific knowledge base by updating all components of the external knowledge base during training. In addition, we introduce an auxiliary training signal to inject more domain-specific knowledge. This auxiliary signal forces RAG-end2end to reconstruct a given sentence by accessing the relevant information from the external knowledge base. Our novel contribution is that, unlike RAG, RAG-end2end does joint training of the retriever and generator for the end QA task and domain adaptation. We evaluate our approach with datasets from three domains: COVID-19, News, and Conversations, and achieve significant performance improvements compared to the original RAG model. Our work has been open-sourced through the HuggingFace Transformers library, attesting to our work’s credibility and technical consistency.","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"55 1","pages":"1-17"},"PeriodicalIF":10.9,"publicationDate":"2022-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"64440765","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 10

FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation 少镜头区域感知机器翻译的基准

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2022-10-01 DOI: 10.1162/tacl_a_00568

Parker Riley, Timothy Dozat, Jan A. Botha, Xavier García, Dan Garrette, Jason Riesa, Orhan Firat, Noah Constant

引用次数: 6

Meta-Learning a Cross-lingual Manifold for Semantic Parsing 元学习——用于语义分析的跨语言流形

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2022-09-26 DOI: 10.1162/tacl_a_00533

Tom Sherborne, Mirella Lapata

引用次数: 7

OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue 面向端到端任务对话的本体感知预训练语言模型

IF 10.9 1区计算机科学

Transactions of the Association for Computational Linguistics Pub Date : 2022-09-10 DOI: 10.1162/tacl_a_00534

Zhi Chen, Yuncong Liu, Lu Chen, Su Zhu, Mengyue Wu, Kai Yu

{"title":"OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue","authors":"Zhi Chen, Yuncong Liu, Lu Chen, Su Zhu, Mengyue Wu, Kai Yu","doi":"10.1162/tacl_a_00534","DOIUrl":"https://doi.org/10.1162/tacl_a_00534","url":null,"abstract":"This paper presents an ontology-aware pretrained language model (OPAL) for end-to-end task-oriented dialogue (TOD). Unlike chit-chat dialogue models, task-oriented dialogue models fulfill at least two task-specific modules: Dialogue state tracker (DST) and response generator (RG). The dialogue state consists of the domain-slot-value triples, which are regarded as the user’s constraints to search the domain-related databases. The large-scale task-oriented dialogue data with the annotated structured dialogue state usually are inaccessible. It prevents the development of the pretrained language model for the task-oriented dialogue. We propose a simple yet effective pretraining method to alleviate this problem, which consists of two pretraining phases. The first phase is to pretrain on large-scale contextual text data, where the structured information of the text is extracted by the information extracting tool. To bridge the gap between the pretraining method and downstream tasks, we design two pretraining tasks: ontology-like triple recovery and next-text generation, which simulates the DST and RG, respectively. The second phase is to fine-tune the pretrained model on the TOD data. The experimental results show that our proposed method achieves an exciting boost and obtains competitive performance even without any TOD data on CamRest676 and MultiWOZ benchmarks.","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"11 1","pages":"68-84"},"PeriodicalIF":10.9,"publicationDate":"2022-09-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44597473","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4