Conference on Empirical Methods in Natural Language Processing最新文献

GPT Deciphering Fedspeak: Quantifying Dissent Among Hawks and Doves GPT 破译联邦言论：量化鹰派和鸽派的不同意见

Conference on Empirical Methods in Natural Language Processing Pub Date : 2024-07-26 DOI: 10.18653/v1/2023.findings-emnlp.434

Denis Peskoff, Adam Visokay, Sander Schulhoff, Benjamin Wachspress, Alan Blinder, Brandon Stewart

引用次数: 0

MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization MAPO：通过模型自适应提示优化提升大型语言模型性能

Conference on Empirical Methods in Natural Language Processing Pub Date : 2024-07-04 DOI: 10.18653/v1/2023.findings-emnlp.215

Yuyan Chen, Zhihao Wen, Ge Fan, Zhengyu Chen, Wei Wu, Dayiheng Liu, Zhixu Li, Bang Liu, Yanghua Xiao

引用次数: 5

HiddenTables and PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies HiddenTables 和 PyQTax：用于 TableQA 的合作游戏和数据集，可确保各种分类标准的规模和数据隐私

Conference on Empirical Methods in Natural Language Processing Pub Date : 2024-06-16 DOI: 10.18653/v1/2023.emnlp-main.442

William Watson, Nicole Cho, T. Balch, Manuela Veloso

{"title":"HiddenTables and PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies","authors":"William Watson, Nicole Cho, T. Balch, Manuela Veloso","doi":"10.18653/v1/2023.emnlp-main.442","DOIUrl":"https://doi.org/10.18653/v1/2023.emnlp-main.442","url":null,"abstract":"A myriad of different Large Language Models (LLMs) face a common challenge in contextually analyzing table question-answering tasks. These challenges are engendered from (1) finite context windows for large tables, (2) multi-faceted discrepancies amongst tokenization patterns against cell boundaries, and (3) various limitations stemming from data confidentiality in the process of using external models such as gpt-3.5-turbo. We propose a cooperative game dubbed\"HiddenTables\"as a potential resolution to this challenge. In essence,\"HiddenTables\"is played between the code-generating LLM\"Solver\"and the\"Oracle\"which evaluates the ability of the LLM agents to solve Table QA tasks. This game is based on natural language schemas and importantly, ensures the security of the underlying data. We provide evidential experiments on a diverse set of tables that demonstrate an LLM's collective inability to generalize and perform on complex queries, handle compositional dependencies, and align natural language to programmatic commands when concrete table schemas are provided. Unlike encoder-based models, we have pushed the boundaries of\"HiddenTables\"to not be limited by the number of rows - therefore we exhibit improved efficiency in prompt and completion tokens. Our infrastructure has spawned a new dataset\"PyQTax\"that spans across 116,671 question-table-answer triplets and provides additional fine-grained breakdowns&labels for varying question taxonomies. Therefore, in tandem with our academic contributions regarding LLMs' deficiency in TableQA tasks,\"HiddenTables\"is a tactile manifestation of how LLMs can interact with massive datasets while ensuring data security and minimizing generation costs.","PeriodicalId":505350,"journal":{"name":"Conference on Empirical Methods in Natural Language Processing","volume":"2 10","pages":"7144-7159"},"PeriodicalIF":0.0,"publicationDate":"2024-06-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141335417","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

DiNeR: A Large Realistic Dataset for Evaluating Compositional Generalization DiNeR：用于评估合成泛化的大型真实数据集

Conference on Empirical Methods in Natural Language Processing Pub Date : 2024-06-07 DOI: 10.18653/v1/2023.emnlp-main.924

ChenGang Hu, Xiao Liu, Yansong Feng

{"title":"DiNeR: A Large Realistic Dataset for Evaluating Compositional Generalization","authors":"ChenGang Hu, Xiao Liu, Yansong Feng","doi":"10.18653/v1/2023.emnlp-main.924","DOIUrl":"https://doi.org/10.18653/v1/2023.emnlp-main.924","url":null,"abstract":"Most of the existing compositional generalization datasets are synthetically-generated, resulting in a lack of natural language variation. While there have been recent attempts to introduce non-synthetic datasets for compositional generalization, they suffer from either limited data scale or a lack of diversity in the forms of combinations. To better investigate compositional generalization with more linguistic phenomena and compositional diversity, we propose the DIsh NamE Recognition (DiNeR) task and create a large realistic Chinese dataset. Given a recipe instruction, models are required to recognize the dish name composed of diverse combinations of food, actions, and flavors. Our dataset consists of 3,811 dishes and 228,114 recipes, and involves plenty of linguistic phenomena such as anaphora, omission and ambiguity. We provide two strong baselines based on T5 and large language models (LLMs). This work contributes a challenging task, baseline methods to tackle the task, and insights into compositional generalization in the context of dish name recognition. Code and data are available at https://github.com/Jumpy-pku/DiNeR.","PeriodicalId":505350,"journal":{"name":"Conference on Empirical Methods in Natural Language Processing","volume":" 6","pages":"14938-14947"},"PeriodicalIF":0.0,"publicationDate":"2024-06-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141370940","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification 聚焦核心：通过剪枝标记压缩实现高效关注，促进文档分类

Conference on Empirical Methods in Natural Language Processing Pub Date : 2024-06-03 DOI: 10.18653/v1/2023.findings-emnlp.909

Jungmin Yun, Mihyeon Kim, Youngbin Kim

{"title":"Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification","authors":"Jungmin Yun, Mihyeon Kim, Youngbin Kim","doi":"10.18653/v1/2023.findings-emnlp.909","DOIUrl":"https://doi.org/10.18653/v1/2023.findings-emnlp.909","url":null,"abstract":"Transformer-based models have achieved dominant performance in numerous NLP tasks. Despite their remarkable successes, pre-trained transformers such as BERT suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including the ones unfavorable to classification performance. To overcome these challenges, we propose integrating two strategies: token pruning and token combining. Token pruning eliminates less important tokens in the attention mechanism's key and value as they pass through the layers. Additionally, we adopt fuzzy logic to handle uncertainty and alleviate potential mispruning risks arising from an imbalanced distribution of each token's importance. Token combining, on the other hand, condenses input sequences into smaller sizes in order to further compress the model. By integrating these two approaches, we not only improve the model's performance but also reduce its computational demands. Experiments with various datasets demonstrate superior performance compared to baseline models, especially with the best improvement over the existing BERT model, achieving +5%p in accuracy and +5.6%p in F1 score. Additionally, memory cost is reduced to 0.61x, and a speedup of 1.64x is achieved.","PeriodicalId":505350,"journal":{"name":"Conference on Empirical Methods in Natural Language Processing","volume":"9 11","pages":"13617-13628"},"PeriodicalIF":0.0,"publicationDate":"2024-06-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141269092","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2

Combining Denoising Autoencoders with Contrastive Learning to fine-tune Transformer Models 将去噪自动编码器与对比学习相结合，对变压器模型进行微调

Conference on Empirical Methods in Natural Language Processing Pub Date : 2024-05-23 DOI: 10.18653/v1/2023.emnlp-main.124

Alejo Lopez-Avila, Víctor Suárez-Paniagua

{"title":"Combining Denoising Autoencoders with Contrastive Learning to fine-tune Transformer Models","authors":"Alejo Lopez-Avila, Víctor Suárez-Paniagua","doi":"10.18653/v1/2023.emnlp-main.124","DOIUrl":"https://doi.org/10.18653/v1/2023.emnlp-main.124","url":null,"abstract":"Recently, using large pretrained Transformer models for transfer learning tasks has evolved to the point where they have become one of the flagship trends in the Natural Language Processing (NLP) community, giving rise to various outlooks such as prompt-based, adapters or combinations with unsupervised approaches, among many others. This work proposes a 3 Phase technique to adjust a base model for a classification task. First, we adapt the model's signal to the data distribution by performing further training with a Denoising Autoencoder (DAE). Second, we adjust the representation space of the output to the corresponding classes by clustering through a Contrastive Learning (CL) method. In addition, we introduce a new data augmentation approach for Supervised Contrastive Learning to correct the unbalanced datasets. Third, we apply fine-tuning to delimit the predefined categories. These different phases provide relevant and complementary knowledge to the model to learn the final task. We supply extensive experimental results on several datasets to demonstrate these claims. Moreover, we include an ablation study and compare the proposed method against other ways of combining these techniques.","PeriodicalId":505350,"journal":{"name":"Conference on Empirical Methods in Natural Language Processing","volume":"50 14","pages":"2021-2032"},"PeriodicalIF":0.0,"publicationDate":"2024-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141103080","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

InsightNet : Structured Insight Mining from Customer Feedback InsightNet：从客户反馈中挖掘结构化洞察力

Conference on Empirical Methods in Natural Language Processing Pub Date : 2024-05-12 DOI: 10.18653/v1/2023.emnlp-industry.53

Sandeep Sricharan Mukku, Manan Soni, Chetan Aggarwal, Jitenkumar Rana, Promod Yenigalla, Rashmi Patange, Shyam Mohan

{"title":"InsightNet : Structured Insight Mining from Customer Feedback","authors":"Sandeep Sricharan Mukku, Manan Soni, Chetan Aggarwal, Jitenkumar Rana, Promod Yenigalla, Rashmi Patange, Shyam Mohan","doi":"10.18653/v1/2023.emnlp-industry.53","DOIUrl":"https://doi.org/10.18653/v1/2023.emnlp-industry.53","url":null,"abstract":"We propose InsightNet, a novel approach for the automated extraction of structured insights from customer reviews. Our end-to-end machine learning framework is designed to overcome the limitations of current solutions, including the absence of structure for identified topics, non-standard aspect names, and lack of abundant training data. The proposed solution builds a semi-supervised multi-level taxonomy from raw reviews, a semantic similarity heuristic approach to generate labelled data and employs a multi-task insight extraction architecture by fine-tuning an LLM. InsightNet identifies granular actionable topics with customer sentiments and verbatim for each topic. Evaluations on real-world customer review data show that InsightNet performs better than existing solutions in terms of structure, hierarchy and completeness. We empirically demonstrate that InsightNet outperforms the current state-of-the-art methods in multi-label topic classification, achieving an F1 score of 0.85, which is an improvement of 11% F1-score over the previous best results. Additionally, InsightNet generalises well for unseen aspects and suggests new topics to be added to the taxonomy.","PeriodicalId":505350,"journal":{"name":"Conference on Empirical Methods in Natural Language Processing","volume":"114 43","pages":"552-566"},"PeriodicalIF":0.0,"publicationDate":"2024-05-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140986216","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Do "English" Named Entity Recognizers Work Well on Global Englishes? 英语 "命名实体识别器在全球英语中效果好吗？

Conference on Empirical Methods in Natural Language Processing Pub Date : 2024-04-20 DOI: 10.18653/v1/2023.findings-emnlp.788

Alexander Shan, John Bauer, Riley Carlson, Christopher D. Manning

{"title":"Do \"English\" Named Entity Recognizers Work Well on Global Englishes?","authors":"Alexander Shan, John Bauer, Riley Carlson, Christopher D. Manning","doi":"10.18653/v1/2023.findings-emnlp.788","DOIUrl":"https://doi.org/10.18653/v1/2023.findings-emnlp.788","url":null,"abstract":"The vast majority of the popular English named entity recognition (NER) datasets contain American or British English data, despite the existence of many global varieties of English. As such, it is unclear whether they generalize for analyzing use of English globally. To test this, we build a newswire dataset, the Worldwide English NER Dataset, to analyze NER model performance on low-resource English variants from around the world. We test widely used NER toolkits and transformer models, including models using the pre-trained contextual models RoBERTa and ELECTRA, on three datasets: a commonly used British English newswire dataset, CoNLL 2003, a more American focused dataset OntoNotes, and our global dataset. All models trained on the CoNLL or OntoNotes datasets experienced significant performance drops-over 10 F1 in some cases-when tested on the Worldwide English dataset. Upon examination of region-specific errors, we observe the greatest performance drops for Oceania and Africa, while Asia and the Middle East had comparatively strong performance. Lastly, we find that a combined model trained on the Worldwide dataset and either CoNLL or OntoNotes lost only 1-2 F1 on both test sets.","PeriodicalId":505350,"journal":{"name":"Conference on Empirical Methods in Natural Language Processing","volume":"115 36","pages":"11778-11791"},"PeriodicalIF":0.0,"publicationDate":"2024-04-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140680893","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Using Interpretation Methods for Model Enhancement 使用解释方法改进模型

Conference on Empirical Methods in Natural Language Processing Pub Date : 2024-04-02 DOI: 10.18653/v1/2023.emnlp-main.28

Zhuo Chen, Chengyue Jiang, Kewei Tu

{"title":"Using Interpretation Methods for Model Enhancement","authors":"Zhuo Chen, Chengyue Jiang, Kewei Tu","doi":"10.18653/v1/2023.emnlp-main.28","DOIUrl":"https://doi.org/10.18653/v1/2023.emnlp-main.28","url":null,"abstract":"In the age of neural natural language processing, there are plenty of works trying to derive interpretations of neural models. Intuitively, when gold rationales exist during training, one can additionally train the model to match its interpretation with the rationales. However, this intuitive idea has not been fully explored. In this paper, we propose a framework of utilizing interpretation methods and gold rationales to enhance models. Our framework is very general in the sense that it can incorporate various interpretation methods. Previously proposed gradient-based methods can be shown as an instance of our framework. We also propose two novel instances utilizing two other types of interpretation methods, erasure/replace-based and extractor-based methods, for model enhancement. We conduct comprehensive experiments on a variety of tasks. Experimental results show that our framework is effective especially in low-resource settings in enhancing models with various interpretation methods, and our two newly-proposed methods outperform gradient-based methods in most settings. Code is available at https://github.com/Chord-Chen-30/UIMER.","PeriodicalId":505350,"journal":{"name":"Conference on Empirical Methods in Natural Language Processing","volume":"25 6","pages":"424-438"},"PeriodicalIF":0.0,"publicationDate":"2024-04-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140753928","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Are Compressed Language Models Less Subgroup Robust? 压缩语言模型的亚群鲁棒性较差吗？

Conference on Empirical Methods in Natural Language Processing Pub Date : 2024-03-26 DOI: 10.18653/v1/2023.emnlp-main.983

Leonidas Gee, Andrea Zugarini, Novi Quadrianto

引用次数: 0