Annual Meeting of the Association for Computational Linguistics最新文献_第3页

On the Efficacy of Sampling Adapters 关于采样适配器的有效性

Annual Meeting of the Association for Computational Linguistics Pub Date : 2023-07-07 DOI: 10.48550/arXiv.2307.03749

Clara Meister, Tiago Pimentel, L. Malagutti, Ethan Gotlieb Wilcox, Ryan Cotterell

{"title":"On the Efficacy of Sampling Adapters","authors":"Clara Meister, Tiago Pimentel, L. Malagutti, Ethan Gotlieb Wilcox, Ryan Cotterell","doi":"10.48550/arXiv.2307.03749","DOIUrl":"https://doi.org/10.48550/arXiv.2307.03749","url":null,"abstract":"Sampling-based decoding strategies are widely employed for generating text from probabilistic models, yet standard ancestral sampling often results in text that is degenerate or incoherent. To alleviate this issue, various modifications to a model’s sampling distribution, such as top-p or top-k sampling, have been introduced and are now ubiquitously used in language generation systems. We propose a unified framework for understanding these techniques, which we term sampling adapters. Sampling adapters often lead to qualitatively better text, which raises the question: From a formal perspective, how are they changing the token-level distributions of language generation models? And why do these local changes lead to higher-quality text? We argue that the shift they enforce can be viewed as a trade-off between precision and recall: while the model loses its ability to produce certain strings, its precision rate on desirable text increases. While this trade-off is not reflected in standard metrics of distribution quality (such as perplexity), we find that several precision-emphasizing measures indeed indicate that sampling adapters can lead to probability distributions more aligned with the true distribution. Further, these measures correlate with higher sequence-level quality scores, specifically, Mauve.","PeriodicalId":352845,"journal":{"name":"Annual Meeting of the Association for Computational Linguistics","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126687084","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Covering Uncommon Ground: Gap-Focused Question Generation for Answer Assessment 涵盖不寻常的领域:针对答案评估的空白问题生成

Annual Meeting of the Association for Computational Linguistics Pub Date : 2023-07-06 DOI: 10.48550/arXiv.2307.03319

Ron Rabin, Alexandre Djerbetian, Roee Engelberg, Lidan Hackmon, G. Elidan, Reut Tsarfaty, A. Globerson

引用次数: 0

InfoSync: Information Synchronization across Multilingual Semi-structured Tables InfoSync:跨多语言半结构化表的信息同步

Annual Meeting of the Association for Computational Linguistics Pub Date : 2023-07-06 DOI: 10.48550/arXiv.2307.03313

Siddharth Khincha, Chelsi Jain, Vivek Gupta, Tushar Kataria, Shuo Zhang

引用次数: 0

Training Models to Generate, Recognize, and Reframe Unhelpful Thoughts 训练产生、识别和重构无用想法的模型

Annual Meeting of the Association for Computational Linguistics Pub Date : 2023-07-06 DOI: 10.48550/arXiv.2307.02768

Mounica Maddela, Megan Ung, Jing Xu, Andrea Madotto, H. Foran, Y-Lan Boureau

引用次数: 2

Generative Zero-Shot Prompt Learning for Cross-Domain Slot Filling with Inverse Prompting 逆提示跨域补槽的生成零射提示学习

Annual Meeting of the Association for Computational Linguistics Pub Date : 2023-07-06 DOI: 10.48550/arXiv.2307.02830

Xuefeng Li, Liwen Wang, Guanting Dong, Keqing He, Jinzheng Zhao, Hao Lei, Jiachi Liu, Weiran Xu

引用次数: 6

KoRC: Knowledge oriented Reading Comprehension Benchmark for Deep Text Understanding 面向知识的深度文本理解阅读理解基准

Annual Meeting of the Association for Computational Linguistics Pub Date : 2023-07-06 DOI: 10.48550/arXiv.2307.03115

Zijun Yao, Yan-Tie Liu, Xin Lv, S. Cao, Jifan Yu, Lei Hou, Juanzi Li

{"title":"KoRC: Knowledge oriented Reading Comprehension Benchmark for Deep Text Understanding","authors":"Zijun Yao, Yan-Tie Liu, Xin Lv, S. Cao, Jifan Yu, Lei Hou, Juanzi Li","doi":"10.48550/arXiv.2307.03115","DOIUrl":"https://doi.org/10.48550/arXiv.2307.03115","url":null,"abstract":"Deep text understanding, which requires the connections between a given document and prior knowledge beyond its text, has been highlighted by many benchmarks in recent years. However, these benchmarks have encountered two major limitations. On the one hand, most of them require human annotation of knowledge, which leads to limited knowledge coverage. On the other hand, they usually use choices or spans in the texts as the answers, which results in narrow answer space. To overcome these limitations, we build a new challenging benchmark named KoRc in this paper. Compared with previous benchmarks, KoRC has two advantages, i.e., broad knowledge coverage and flexible answer format. Specifically, we utilize massive knowledge bases to guide annotators or large language models (LLMs) to construct knowledgable questions. Moreover, we use labels in knowledge bases rather than spans or choices as the final answers. We test state-of-the-art models on KoRC and the experimental results show that the strongest baseline only achieves 68.3% and 30.0% F1 measure in the in-distribution and out-of-distribution test set, respectively. These results indicate that deep text understanding is still an unsolved challenge. The benchmark dataset, leaderboard, and baseline methods are released in https://github.com/THU-KEG/KoRC.","PeriodicalId":352845,"journal":{"name":"Annual Meeting of the Association for Computational Linguistics","volume":"23 4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128167653","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2

NatLogAttack: A Framework for Attacking Natural Language Inference Models with Natural Logic NatLogAttack:利用自然逻辑攻击自然语言推理模型的框架

Annual Meeting of the Association for Computational Linguistics Pub Date : 2023-07-06 DOI: 10.48550/arXiv.2307.02849

Zi'ou Zheng, Xiaodan Zhu

{"title":"NatLogAttack: A Framework for Attacking Natural Language Inference Models with Natural Logic","authors":"Zi'ou Zheng, Xiaodan Zhu","doi":"10.48550/arXiv.2307.02849","DOIUrl":"https://doi.org/10.48550/arXiv.2307.02849","url":null,"abstract":"Reasoning has been a central topic in artificial intelligence from the beginning. The recent progress made on distributed representation and neural networks continues to improve the state-of-the-art performance of natural language inference. However, it remains an open question whether the models perform real reasoning to reach their conclusions or rely on spurious correlations. Adversarial attacks have proven to be an important tool to help evaluate the Achilles’ heel of the victim models. In this study, we explore the fundamental problem of developing attack models based on logic formalism. We propose NatLogAttack to perform systematic attacks centring around natural logic, a classical logic formalism that is traceable back to Aristotle’s syllogism and has been closely developed for natural language inference. The proposed framework renders both label-preserving and label-flipping attacks.We show that compared to the existing attack models, NatLogAttack generates better adversarial examples with fewer visits to the victim models. The victim models are found to be more vulnerable under the label-flipping setting. NatLogAttack provides a tool to probe the existing and future NLI models’ capacity from a key viewpoint and we hope more logic-based attacks will be further explored for understanding the desired property of reasoning.","PeriodicalId":352845,"journal":{"name":"Annual Meeting of the Association for Computational Linguistics","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133290461","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training BLEURT具有通用翻译:基于最小风险训练的自动度量分析

Annual Meeting of the Association for Computational Linguistics Pub Date : 2023-07-06 DOI: 10.48550/arXiv.2307.03131

Yiming Yan, Tao Wang, Chengqi Zhao, Shujian Huang, Jiajun Chen, Mingxuan Wang

{"title":"BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training","authors":"Yiming Yan, Tao Wang, Chengqi Zhao, Shujian Huang, Jiajun Chen, Mingxuan Wang","doi":"10.48550/arXiv.2307.03131","DOIUrl":"https://doi.org/10.48550/arXiv.2307.03131","url":null,"abstract":"Automatic metrics play a crucial role in machine translation. Despite the widespread use of n-gram-based metrics, there has been a recent surge in the development of pre-trained model-based metrics that focus on measuring sentence semantics. However, these neural metrics, while achieving higher correlations with human evaluations, are often considered to be black boxes with potential biases that are difficult to detect. In this study, we systematically analyze and compare various mainstream and cutting-edge automatic metrics from the perspective of their guidance for training machine translation systems. Through Minimum Risk Training (MRT), we find that certain metrics exhibit robustness defects, such as the presence of universal adversarial translations in BLEURT and BARTScore. In-depth analysis suggests two main causes of these robustness deficits: distribution biases in the training datasets, and the tendency of the metric paradigm. By incorporating token-level constraints, we enhance the robustness of evaluation metrics, which in turn leads to an improvement in the performance of machine translation systems. Codes are available at https://github.com/powerpuffpomelo/fairseq_mrt.","PeriodicalId":352845,"journal":{"name":"Annual Meeting of the Association for Computational Linguistics","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124299750","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 5

UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language 麒麟:统一认知信号重建，连接认知信号和人类语言

Annual Meeting of the Association for Computational Linguistics Pub Date : 2023-07-06 DOI: 10.48550/arXiv.2307.05355

Nuwa Xi, Sendong Zhao, Hao Wang, Chi Liu, Bing Qin, Ting Liu

{"title":"UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language","authors":"Nuwa Xi, Sendong Zhao, Hao Wang, Chi Liu, Bing Qin, Ting Liu","doi":"10.48550/arXiv.2307.05355","DOIUrl":"https://doi.org/10.48550/arXiv.2307.05355","url":null,"abstract":"Decoding text stimuli from cognitive signals (e.g. fMRI) enhances our understanding of the human language system, paving the way for building versatile Brain-Computer Interface. However, existing studies largely focus on decoding individual word-level fMRI volumes from a restricted vocabulary, which is far too idealized for real-world application. In this paper, we propose fMRI2text, the first open-vocabulary task aiming to bridge fMRI time series and human language. Furthermore, to explore the potential of this new task, we present a baseline solution, UniCoRN: the Unified Cognitive Signal ReconstructioN for Brain Decoding. By reconstructing both individual time points and time series, UniCoRN establishes a robust encoder for cognitive signals (fMRI & EEG). Leveraging a pre-trained language model as decoder, UniCoRN proves its efficacy in decoding coherent text from fMRI series across various split settings. Our model achieves a 34.77% BLEU score on fMRI2text, and a 37.04% BLEU when generalized to EEG-to-text decoding, thereby surpassing the former baseline. Experimental results indicate the feasibility of decoding consecutive fMRI volumes, and the effectiveness of decoding different cognitive signals using a unified structure.","PeriodicalId":352845,"journal":{"name":"Annual Meeting of the Association for Computational Linguistics","volume":"12 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126798918","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social Relationships 你的配偶需要专业帮助:通过模拟社会关系来确定信息的语境适当性

Annual Meeting of the Association for Computational Linguistics Pub Date : 2023-07-06 DOI: 10.48550/arXiv.2307.02763

David Jurgens, Agrima Seth, Jack E. Sargent, Athena Aghighi, Michael Geraci

{"title":"Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social Relationships","authors":"David Jurgens, Agrima Seth, Jack E. Sargent, Athena Aghighi, Michael Geraci","doi":"10.48550/arXiv.2307.02763","DOIUrl":"https://doi.org/10.48550/arXiv.2307.02763","url":null,"abstract":"Understanding interpersonal communication requires, in part, understanding the social context and norms in which a message is said. However, current methods for identifying offensive content in such communication largely operate independent of context, with only a few approaches considering community norms or prior conversation as context. Here, we introduce a new approach to identifying inappropriate communication by explicitly modeling the social relationship between the individuals. We introduce a new dataset of contextually-situated judgments of appropriateness and show that large language models can readily incorporate relationship information to accurately identify appropriateness in a given context. Using data from online conversations and movie dialogues, we provide insight into how the relationships themselves function as implicit norms and quantify the degree to which context-sensitivity is needed in different conversation settings. Further, we also demonstrate that contextual-appropriateness judgments are predictive of other social factors expressed in language such as condescension and politeness.","PeriodicalId":352845,"journal":{"name":"Annual Meeting of the Association for Computational Linguistics","volume":"335 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133694955","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0