Natural Language Processing Journal最新文献_第10页

Out-of-vocabulary word embedding learning based on reading comprehension mechanism 基于阅读理解机制的词汇外嵌入学习

Natural Language Processing Journal Pub Date : 2023-10-31 DOI: 10.1016/j.nlp.2023.100038

Zhongyu Zhuang , Ziran Liang , Yanghui Rao , Haoran Xie , Fu Lee Wang

{"title":"Out-of-vocabulary word embedding learning based on reading comprehension mechanism","authors":"Zhongyu Zhuang , Ziran Liang , Yanghui Rao , Haoran Xie , Fu Lee Wang","doi":"10.1016/j.nlp.2023.100038","DOIUrl":"https://doi.org/10.1016/j.nlp.2023.100038","url":null,"abstract":"<div><p>Currently, most natural language processing tasks use word embeddings as the representation of words. However, when encountering out-of-vocabulary (OOV) words, the performance of downstream models that use word embeddings as input is often quite limited. To solve this problem, the latest methods mainly infer the meaning of OOV words based on two types of information sources: the morphological structure of OOV words and the contexts in which they appear. However, the low frequency of OOV words themselves usually makes them difficult to learn in pre-training tasks by general word embedding models. In addition, this characteristic of OOV word embedding learning also brings the problem of context scarcity. Therefore, we introduce the concept of “similar contexts” based on the classical “distributed hypothesis” in linguistics, by borrowing from the human reading comprehension mechanisms to make up for the deficiency of insufficient contexts in previous OOV word embedding learning work. The experimental results show that our model achieved the highest relative scores in both intrinsic and extrinsic evaluation tasks, which demonstrates the positive effect of the “similar contexts” introduced in our model on OOV word embedding learning.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"5 ","pages":"Article 100038"},"PeriodicalIF":0.0,"publicationDate":"2023-10-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2949719123000353/pdfft?md5=3f3a30c80249e275dc7207c19446fc4a&pid=1-s2.0-S2949719123000353-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"91987778","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

On the instability of further pre-training: Does a single sentence matter to BERT? 关于进一步预训练的不稳定性:单个句子对BERT有影响吗?

Natural Language Processing Journal Pub Date : 2023-10-27 DOI: 10.1016/j.nlp.2023.100037

Luca Bacco , Gosse Minnema , Tommaso Caselli , Felice Dell’Orletta , Mario Merone , Malvina Nissim

{"title":"On the instability of further pre-training: Does a single sentence matter to BERT?","authors":"Luca Bacco , Gosse Minnema , Tommaso Caselli , Felice Dell’Orletta , Mario Merone , Malvina Nissim","doi":"10.1016/j.nlp.2023.100037","DOIUrl":"https://doi.org/10.1016/j.nlp.2023.100037","url":null,"abstract":"<div><p>We observe a remarkable instability in BERT-like models: minimal changes in the internal representations of BERT, as induced by one-step further pre-training with even a single sentence, can noticeably change the behaviour of subsequently fine-tuned models. While the pre-trained models seem to be essentially the same, also by means of established similarity assessment techniques, the measurable tiny changes appear to substantially impact the models’ tuning path, leading to significantly different fine-tuned systems and affecting downstream performance. After testing a very large number of combinations, which we briefly summarize, the experiments reported in this short paper focus on an intermediate phase consisting of a single-step and single-sentence masked language modeling stage and its impact on a sentiment analysis task. We discuss a series of unexpected findings which leave some open questions over the nature and stability of further pre-training.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"5 ","pages":"Article 100037"},"PeriodicalIF":0.0,"publicationDate":"2023-10-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2949719123000341/pdfft?md5=ea3c6f4e3559eae8be4a84b5fe77fb85&pid=1-s2.0-S2949719123000341-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"92122458","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Knowledge representation and acquisition in the era of large language models: Reflections on learning to reason via PAC-Semantics 大语言模型时代的知识表示与获取:通过pac语义学习推理的思考

Natural Language Processing Journal Pub Date : 2023-10-25 DOI: 10.1016/j.nlp.2023.100036

Ionela G. Mocanu, Vaishak Belle

{"title":"Knowledge representation and acquisition in the era of large language models: Reflections on learning to reason via PAC-Semantics","authors":"Ionela G. Mocanu, Vaishak Belle","doi":"10.1016/j.nlp.2023.100036","DOIUrl":"https://doi.org/10.1016/j.nlp.2023.100036","url":null,"abstract":"<div><p>Human beings are known for their remarkable ability to comprehend, analyse, and interpret common sense knowledge. This ability is critical for exhibiting intelligent behaviour, often defined as a mapping from beliefs to actions, which has led to attempts to formalize and capture explicit representations in the form of databases, knowledge bases, and ontologies in AI agents.</p><p>But in the era of large language models (LLMs), this emphasis might seem unnecessary. After all, these models already capture the extent of human knowledge and can infer appropriate things from it (presumably) as per some innate logical rules. The question then is whether they can also be trained to perform mathematical computations.</p><p>Although the consensus on the reliability of such models is still being studied, early results do seem to suggest they do not offer logically and mathematically consistent results. In this short summary article, we articulate the motivations for still caring about logical/symbolic artefacts and representations, and report on recent progress in learning to reason via the so-called probably approximately correct (PAC)-semantics.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"5 ","pages":"Article 100036"},"PeriodicalIF":0.0,"publicationDate":"2023-10-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S294971912300033X/pdfft?md5=bf1ac9b507bf03d01852d71158a672d4&pid=1-s2.0-S294971912300033X-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"91987777","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Multi-aspect attentive text representations for simple question answering over knowledge base 基于知识库的简单问答的多方面注意文本表示

Natural Language Processing Journal Pub Date : 2023-10-05 DOI: 10.1016/j.nlp.2023.100035

Zhixiang Zeng , Yuefeng Li , Jianming Yong , Xiaohui Tao , Vicky Liu

{"title":"Multi-aspect attentive text representations for simple question answering over knowledge base","authors":"Zhixiang Zeng , Yuefeng Li , Jianming Yong , Xiaohui Tao , Vicky Liu","doi":"10.1016/j.nlp.2023.100035","DOIUrl":"https://doi.org/10.1016/j.nlp.2023.100035","url":null,"abstract":"<div><p>With the deepening of knowledge base research and application, question answering over knowledge base, also called KBQA, has recently received more and more attention from researchers. Most previous KBQA models focus on mapping the input query and the fact in KBs into an embedding format. Then the similarity between the query vector and the fact vector is computed eventually. Based on the similarity, each query can obtain an answer representing a tuple (subject, predicate, object) from the KBs. However, the information about each word in the input question will lose inevitably during the process. To retain as much original information as possible, we introduce an attention-based recurrent neural network model with interactive similarity matrixes. It can extract more comprehensive information from the hierarchical structure of words among queries and tuples stored in the knowledge base. This work makes three main contributions: (1) A neural network-based question-answering model for the knowledge base is proposed to handle single relation questions. (2) An attentive module is designed to obtain information from multiple aspects to represent queries and data, which contributes to avoiding losing potentially valuable information. (3) Similarity matrixes are introduced to obtain the interaction information between queries and data from the knowledge base. Experimental results show that our proposed model performs better on simple questions than state-of-the-art in several effectiveness measures.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"5 ","pages":"Article 100035"},"PeriodicalIF":0.0,"publicationDate":"2023-10-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50194379","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A Bidirectional LSTM approach for written script auto evaluation using keywords-based pattern matching 一种基于关键字的模式匹配的双向LSTM脚本自动评估方法

Natural Language Processing Journal Pub Date : 2023-09-15 DOI: 10.1016/j.nlp.2023.100033

Prabakaran N. , Kannadasan R. , Krishnamoorthy A. , Vijay Kakani

{"title":"A Bidirectional LSTM approach for written script auto evaluation using keywords-based pattern matching","authors":"Prabakaran N. , Kannadasan R. , Krishnamoorthy A. , Vijay Kakani","doi":"10.1016/j.nlp.2023.100033","DOIUrl":"https://doi.org/10.1016/j.nlp.2023.100033","url":null,"abstract":"<div><p>The evaluation process necessitates significant work in order to effectively and impartially assess the growing number of new subjects and interests in courses. This paper aims at auto-evaluating and setting scores for individuals similar to those provided by humans using deep learning models. This system is built purely to decipher the English characters and numbers from images, convert them into text format, and match the existing written scripts or custom keywords provided by the invigilators to check the answers. The Handwritten Text Recognition (HTR) model fervors and implements an algorithm that is capable of evaluating written scripts based on handwriting and comparing it with the custom keywords provided, whereas the existing models using Convolutional Neural networks (CNN) or Recurrent Neural networks (RNN) suffer from the Vanishing Gradient problem. The core objective of this model is to reduce manual paper checking using Bidirectional Long Short Term Memory (BiLSTM) and CRNN (Convolutional Recurrent Neural Networks). It has been implemented more than the models built on conventional approaches in aspects of performance, efficiency, and better text recognition. The inputs given to the model are in the form of custom keywords; the system processes them through HTR and image processing techniques of segmentation; and the output formats the percentage obtained by the student, word error rate, number of words misspelt, synonyms produced, and the effective outcome. The system has the capability to identify and highlight errors made by students. This feature is advantageous for both students and teachers, as it saves a significant amount of time. Even if the keywords used by students do not align perfectly, the advanced processing models employed by the system possess the intelligence to provide a reasonable number of marks.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"5 ","pages":"Article 100033"},"PeriodicalIF":0.0,"publicationDate":"2023-09-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50194422","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

GPT-3.5, GPT-4, or BARD? Evaluating LLMs reasoning ability in zero-shot setting and performance boosting through prompts GPT-3.5、GPT-4还是BARD？零样本环境下LLM推理能力评估及提示提升

Natural Language Processing Journal Pub Date : 2023-09-14 DOI: 10.1016/j.nlp.2023.100032

Jessica López Espejel, El Hassane Ettifouri, Mahaman Sanoussi Yahaya Alassan, El Mehdi Chouham, Walid Dahhane

{"title":"GPT-3.5, GPT-4, or BARD? Evaluating LLMs reasoning ability in zero-shot setting and performance boosting through prompts","authors":"Jessica López Espejel, El Hassane Ettifouri, Mahaman Sanoussi Yahaya Alassan, El Mehdi Chouham, Walid Dahhane","doi":"10.1016/j.nlp.2023.100032","DOIUrl":"https://doi.org/10.1016/j.nlp.2023.100032","url":null,"abstract":"<div><p>Large Language Models (LLMs) have exhibited remarkable performance on various Natural Language Processing (NLP) tasks. However, there is a current hot debate regarding their reasoning capacity. In this paper, we examine the performance of GPT-3.5, GPT-4, and BARD models, by performing a thorough technical evaluation on different reasoning tasks across eleven distinct datasets. Our paper provides empirical evidence showcasing the superior performance of ChatGPT-4 in comparison to both ChatGPT-3.5 and BARD in zero-shot setting throughout almost all evaluated tasks. While the superiority of GPT-4 compared to GPT-3.5 might be explained by its larger size and NLP efficiency, this was not evident for BARD. We also demonstrate that the three models show limited proficiency in Inductive, Mathematical, and Multi-hop Reasoning Tasks. To bolster our findings, we present a detailed and comprehensive analysis of the results from these three models. Furthermore, we propose a set of engineered prompts that enhances the zero-shot setting performance of all three models.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"5 ","pages":"Article 100032"},"PeriodicalIF":0.0,"publicationDate":"2023-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50194423","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

An evaluation on large language model outputs: Discourse and memorization 大型语言模型输出的评价：语篇与记忆

Natural Language Processing Journal Pub Date : 2023-09-01 DOI: 10.1016/j.nlp.2023.100024

Adrian de Wynter , Xun Wang , Alex Sokolov , Qilong Gu , Si-Qing Chen

引用次数: 7

Natural Language Processing Journal Pub Date : 2023-09-01 DOI: 10.1016/j.nlp.2023.100029

Kazim Ali Mazhar , Matthias Brodtbeck , Gabriele Gühring

{"title":"Similarity learning of product descriptions and images using multimodal neural networks","authors":"Kazim Ali Mazhar , Matthias Brodtbeck , Gabriele Gühring","doi":"10.1016/j.nlp.2023.100029","DOIUrl":"https://doi.org/10.1016/j.nlp.2023.100029","url":null,"abstract":"<div><p>Multimodal deep learning is an emerging research topic in machine learning and involves the parallel processing of different modalities of data such as texts, images and audiovisual data. Well-known application areas are multimodal image and video processing as well as speech recognition. In this paper, we propose a multimodal neural network that measures the similarity of text-written product descriptions and images and has applications in inventory reconciliation and search engine optimization. We develop two models. The first takes image and text data, each processed by convolutional neural networks, and combines the two modalities. The second is based on a bidirectional triplet loss function. We conduct experiments using <strong>ABO!</strong> (<strong>ABO!</strong>) dataset and an industry-related dataset used for the inventory reconciliation of a mechanical engineering company. Our first model achieves an accuracy of 92.37% with ResNet152 on the <strong>ABO!</strong> dataset and 99.11% with MobileNetV3_Large on our industry-related dataset. By extending this model to a model with three inputs, two text inputs and one image input, we greatly improve the performance and achieve an accuracy of 97.57% on the <strong>ABO!</strong> dataset and 99.83% with our industry related inventory dataset. Our second model based on the triplet loss achieves only an accuracy of 73.85% on the <strong>ABO!</strong> dataset. However, our experiments demonstrate that multimodal networks consistently perform better when measuring the similarity of products, even in situations where one modality lacks sufficient data, because it is complemented with the other modality. Our proposed approaches open up several possibilities for further optimization of search engines.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"4 ","pages":"Article 100029"},"PeriodicalIF":0.0,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50187585","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Sentimental Contrastive Learning for event representation 事件表征的情感对比学习

Natural Language Processing Journal Pub Date : 2023-09-01 DOI: 10.1016/j.nlp.2023.100031

Yan Zhou, Xiaodong Li

{"title":"Sentimental Contrastive Learning for event representation","authors":"Yan Zhou, Xiaodong Li","doi":"10.1016/j.nlp.2023.100031","DOIUrl":"https://doi.org/10.1016/j.nlp.2023.100031","url":null,"abstract":"<div><p>Event representation learning is crucial for numerous event-driven tasks, as the quality of event representations greatly influences the performance of these tasks. However, many existing event representation methods exhibit a heavy reliance on semantic features, often neglecting the wealth of information available in other dimensions of events. Consequently, these methods struggle to capture subtle distinctions between events. Incorporating sentimental information can be particularly useful when modeling event data, as leveraging such information can yield superior event representations. To effectively integrate sentimental information, we propose a novel event representation learning framework, namely <strong>S</strong>entimental <strong>C</strong>ontrastive <strong>L</strong>earning (<strong>SCL</strong>). Specifically, we firstly utilize BERT as the backbone network for pre-training and obtain the initial event representations. Subsequently, we employ instance-level and cluster-level contrastive learning to fine-tune the original event representations. We introduce two distinct contrastive losses respectively for instance-level and cluster-level contrastive learning, each aiming to incorporate sentimental information from different perspectives. To evaluate the effectiveness of our proposed model, we select the event similarity evaluation task and conduct experiments on three representative datasets. Extensive experimental results demonstrate obvious performance improvement achieved by our approach over many other models.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"4 ","pages":"Article 100031"},"PeriodicalIF":0.0,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50187586","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Bipol: A novel multi-axes bias evaluation metric with explainability for NLP Bipol：一种新的可解释NLP的多轴偏差评估度量

Natural Language Processing Journal Pub Date : 2023-09-01 DOI: 10.1016/j.nlp.2023.100030

Lama Alkhaled , Tosin Adewumi , Sana Sabah Sabry

引用次数: 0