Natural Language Processing Journal最新文献

{"title":"Towards effective teaching assistants: From intent-based chatbots to LLM-powered teaching assistants","authors":"Bashaer Alsafari ,&nbsp;Eric Atwell ,&nbsp;Aisha Walker ,&nbsp;Martin Callaghan","doi":"10.1016/j.nlp.2024.100101","DOIUrl":"10.1016/j.nlp.2024.100101","url":null,"abstract":"<div><p>As chatbot technology undergoes a transformative phase in the era of artificial intelligence (AI), the integration of advanced AI models emerges as a focal point for reshaping conversational agents within the education sector. This paper explores the evolution of educational chatbot development, specifically focusing on building a teaching assistant for Data Mining and Text Analytics courses at the University of Leeds. The primary objective is to investigate and compare traditional intent-based chatbot approaches with the advanced retrieval-augmented generation (RAG) method, aiming to improve the efficiency and adaptability of teaching assistants in higher education. The study begins with the development of an Amazon Alexa teaching skill, assessing the efficacy of traditional chatbot development in higher education. To enrich the chatbot knowledge base, the research then employs an automated question–answer generation (QAG) approach using the QG Lumos Learning tool to extract contextually grounded question–answer datasets from course materials. Subsequently, the RAG-based system is proposed, leveraging LangChain with the OpenAI GPT-3.5 Turbo model. Findings highlight limitations in intent-based approaches, emphasising the need for more adaptive solutions. The proposed RAG-based teaching assistant demonstrates significant improvements in efficiently handling diverse queries, representing a paradigm shift in educational chatbot capabilities. These findings provide an in-depth understanding of the development phase, specifically illustrating the impact on chatbot performance by contrasting traditional methods with large language model-based approaches. The study contributes valuable perspectives on enhancing adaptability and effectiveness in AI-powered educational tools, providing essential considerations for future developments in the field.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"8 ","pages":"Article 100101"},"PeriodicalIF":0.0,"publicationDate":"2024-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2949719124000499/pdfft?md5=7de3208cd4d6adf93098711dcb0bb283&pid=1-s2.0-S2949719124000499-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142162994","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A combined AraBERT and Voting Ensemble classifier model for Arabic sentiment analysis","authors":"Dhaou Ghoul ,&nbsp;Jérémy Patrix ,&nbsp;Gaël Lejeune ,&nbsp;Jérôme Verny","doi":"10.1016/j.nlp.2024.100100","DOIUrl":"10.1016/j.nlp.2024.100100","url":null,"abstract":"<div><p>For sentiment analysis of short texts (e.g. movie reviews, tweets, etc.), one approach is to build machine learning models that can determine their tones (positive, negative, neutral). However, these natural language processing (NLP) studies are missing when there is a lack of high-quality and large-scale training data for specific languages such as Arabic. In this paper, we present three machine learning models designed to classify sentiment Arabic tweets developed for a Kaggle competition. We present a Voting Ensemble classifier taking advantage of both character-level and word-level features. We also propose an AraBERT (Arabic Bidirectional Encoder Representations from Transformers) model with preprocessing using Farasa Segmenter. Finally, we combine these first two approaches as a third approach (Voting Ensemble classifier using AraBERT embeddings). Performance measures of results show improvement over previous efforts for all models. The third model exhibits strong performance with a 73.98% F-score score. The work presented here could be useful for future studies and for new Arabic sentiment analysis online services or competitions.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"8 ","pages":"Article 100100"},"PeriodicalIF":0.0,"publicationDate":"2024-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2949719124000487/pdfft?md5=0cdd68616cd0023e6f056de98e086b2d&pid=1-s2.0-S2949719124000487-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142274873","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A novel prompting method for few-shot NER via LLMs","authors":"Qi Cheng ,&nbsp;Liqiong Chen ,&nbsp;Zhixing Hu ,&nbsp;Juan Tang ,&nbsp;Qiang Xu ,&nbsp;Binbin Ning","doi":"10.1016/j.nlp.2024.100099","DOIUrl":"10.1016/j.nlp.2024.100099","url":null,"abstract":"<div><p>In various natural language processing tasks, significant strides have been made by Large Language Models (LLMs). Researchers leverage prompt method to conduct LLMs in accomplishing specific tasks under few-shot conditions. However, the prevalent use of LLMs’ prompt methods mainly focuses on guiding generative tasks, and employing existing prompts may result in poor performance in Named Entity Recognition (NER) tasks. To tackle this challenge, we propose a novel prompting method for few-shot NER. By enhancing existing prompt methods, we devise a standardized prompts tailored for the utilization of LLMs in NER tasks. Specifically, we structure the prompts into three components: task definition, few-shot demonstration, and output format. The task definition conducts LLMs in performing NER tasks, few-shot demonstration assists LLMs in understanding NER task objectives through specific output demonstration, and output format restricts LLMs’ output to prevent the generation of unnecessary results. The content of these components has been specifically tailored for NER tasks. Moreover, for the few-shot demonstration within the prompts, we propose a selection strategy that utilizes feedback from LLMs’ outputs to identify more suitable few-shot demonstration as prompts. Additionally, to enhance entity recognition performance, we enrich the prompts by summarizing error examples from the output process of LLMs and integrating them as additional prompts.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"8 ","pages":"Article 100099"},"PeriodicalIF":0.0,"publicationDate":"2024-08-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2949719124000475/pdfft?md5=e7e56213f461ce5ea69e8b3be1581d14&pid=1-s2.0-S2949719124000475-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142076491","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"HarmonyNet: Navigating hate speech detection","authors":"Shaina Raza,&nbsp;Veronica Chatrath","doi":"10.1016/j.nlp.2024.100098","DOIUrl":"10.1016/j.nlp.2024.100098","url":null,"abstract":"<div><p>In the digital era, social media platforms have become central to communication across various domains. However, the vast spread of unregulated content often leads to the prevalence of hate speech and toxicity. Existing methods to detect this toxicity struggle with context sensitivity, accommodating diverse dialects, and adapting to varied communication styles. To tackle these challenges, we introduce an ensemble classifier that leverages the strengths of language models and traditional deep neural network architectures for more effective hate speech detection on social media. Our evaluations show that this hybrid approach outperforms individual models and exhibits robustness against adversarial attacks. Future efforts will aim to enhance the model’s architecture to further boost its efficiency and extend its capability to recognize hate speech across an even wider range of languages and dialects.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"8 ","pages":"Article 100098"},"PeriodicalIF":0.0,"publicationDate":"2024-08-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2949719124000463/pdfft?md5=a4006c27711b7b1ab993698b402c7e9e&pid=1-s2.0-S2949719124000463-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142076492","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Kurdish end-to-end speech synthesis using deep neural networks","authors":"Sabat Salih Muhamad ,&nbsp;Hadi Veisi ,&nbsp;Aso Mahmudi ,&nbsp;Abdulhady Abas Abdullah ,&nbsp;Farhad Rahimi","doi":"10.1016/j.nlp.2024.100096","DOIUrl":"10.1016/j.nlp.2024.100096","url":null,"abstract":"<div><p>This article introduces an end-to-end text-to-speech (TTS) system for the low-resourced language of Central Kurdish (CK, also known as Sorani) and tackles the challenges associated with limited data availability. We have compiled a dataset suitable for end-to-end text-to-speech that includes 21 h of CK female voice paired with corresponding texts. To identify the optimal performing system, we employed Tacotron2, an end-to-end deep neural network for speech synthesis, in three training experiments. The process involves training Tacotron2 using a pre-trained English system, followed by training two models from scratch with full and intonationally balanced datasets. We evaluated the effectiveness of these models using Mean Opinion Score (MOS), a subjective evaluation metric. Our findings demonstrate that the model trained from scratch on the full CK dataset surpasses both the model trained with the intonationally balanced dataset and the model trained using a pre-trained English model in terms of naturalness and intelligibility by achieving a MOS of 4.78 out of 5.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"8 ","pages":"Article 100096"},"PeriodicalIF":0.0,"publicationDate":"2024-08-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S294971912400044X/pdfft?md5=1041be1fe8e6d421c55a8b57704a5308&pid=1-s2.0-S294971912400044X-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142040515","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Analyzing public sentiment on sustainability: A comprehensive review and application of sentiment analysis techniques","authors":"Tess Anderson,&nbsp;Sayani Sarkar,&nbsp;Robert Kelley","doi":"10.1016/j.nlp.2024.100097","DOIUrl":"10.1016/j.nlp.2024.100097","url":null,"abstract":"<div><p>In the contemporary context of escalating environmental concerns, understanding public sentiment toward sustainability initiatives is crucial for shaping effective policies and practices. This research explores the efficacy of sentiment analysis in mining social media data to gauge public attitudes toward sustainability efforts. This study employs a variety of machine learning and deep learning models to perform sentiment analysis utilizing a dataset comprising tweets related to human perception towards environmental sustainability. The aim is to transform unstructured social media text into structured sentiment scores. The comparative analysis includes pre-trained sentiment analysis models like VADER, TextBlob, and Flair with traditional machine learning models such as Logistic Regression, SVM, Decision Tree, Naive Bayes, Random Forest, alongside advanced deep learning techniques like LSTM and pre-trained models BERT and GPT-2. Our results reveal significant variations in model performance, underscoring the importance of selecting appropriate sentiment analysis tools that align with the nuanced domain of sustainability. The study further emphasizes the role of transparent and reproducible research practices in advancing trustworthy AI applications. By providing insights into public opinions on sustainability, this research contributes to the broader discourse on leveraging AI to foster environmental responsibility and action. This work not only illustrates the potential of sentiment analysis in understanding public discourse but also highlights the critical need for tailored approaches that consider the specificity of the sustainability context.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"8 ","pages":"Article 100097"},"PeriodicalIF":0.0,"publicationDate":"2024-08-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2949719124000451/pdfft?md5=91b21220bcf98cfd936b66f143eeabb2&pid=1-s2.0-S2949719124000451-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141985211","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Scholarly recommendation system for NIH funded grants based on biomedical word embedding models","authors":"Zitong Zhang,&nbsp;Ashraf Yaseen,&nbsp;Hulin Wu","doi":"10.1016/j.nlp.2024.100095","DOIUrl":"10.1016/j.nlp.2024.100095","url":null,"abstract":"<div><h3>Objective:</h3><p>Research grants, which are available from several sources, are essential for scholars to sustain a good standing in academia. Although securing grant funds for research is very competitive, being able to locate and find previously funded grants and projects that are relevant to researchers’ interests would be very helpful. In this work, we developed a funded-grants/projects recommendation system for the National Institute of Health (NIH) grants.</p></div><div><h3>Methods:</h3><p>Our system aims to recommend funded grants to researchers based on their publications or input keywords. By extracting summary information from funded grants and their associated applications, we employed two embedding models for biomedical words and sentences (biowordvec and biosentvec), and compare multiple recommendation methods to recommend the most relevant funded grants for researchers’ input</p></div><div><h3>Results:</h3><p>Compared to a baseline method, the recommendation system based on biomedical word embedding models provided higher performance. The system also received an average rate of 3.53 out of 5, based on the relevancy evaluation results from biomedical researchers.</p></div><div><h3>Conclusion:</h3><p>Both internal and external evaluation results prove the effectiveness of our recommendation system. The system would be helpful for biomedical researchers to locate and find previously funded grants related to their interests.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"8 ","pages":"Article 100095"},"PeriodicalIF":0.0,"publicationDate":"2024-08-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2949719124000438/pdfft?md5=0a103c48dd28f4ddba9599863fa7dfc2&pid=1-s2.0-S2949719124000438-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141964605","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Automatic language ability assessment method based on natural language processing","authors":"Nonso Nnamoko ,&nbsp;Themis Karaminis ,&nbsp;Jack Procter ,&nbsp;Joseph Barrowclough ,&nbsp;Ioannis Korkontzelos","doi":"10.1016/j.nlp.2024.100094","DOIUrl":"10.1016/j.nlp.2024.100094","url":null,"abstract":"<div><h3>Background and Objectives:</h3><p>The Wechsler Abbreviated Scales of Intelligence second edition (WASI-II) is a standardised assessment tool that is widely used to assess cognitive ability in clinical, research, and educational settings. In one of the components of this assessment, referred to as the Vocabulary task, the assessed individuals are presented with words (called stimulus items), and asked to explain what each word mean. Their responses are hand-scored based on a list of pre-rated sample responses [0-Point (poor), 1-Point (moderate), or 2-Point (excellent)] that is provided in the accompanying manual of WASI-II. This scoring method is time-consuming, and scoring of responses that do not fully match the pre-rated ones may vary between individual scorers. In this study, we aim to use natural language processing techniques to automate the scoring procedure and make it more time-efficient and reliable (objective).</p></div><div><h3>Methods:</h3><p>Utilising five different word embeddings (Word2vec, Global Vectors, Bidirectional Encoder Representations from Transformers, Generative Pre-trained Transformer 2, and Embeddings from Language Model), we transformed stimulus items and pre-rated responses from the WASI-II Vocabulary task into machine-readable vectors. We measured distance with cosine similarity, evaluating each model against a rational-expectations hypothesis that vector representations for stimuli should align closely with 2-Point responses and diverge from 0-Point responses. Assessment involved frequency of consistent representation and the Pearson correlation coefficient, examining overall consistency with the manual’s ranking across all items and sample responses.</p></div><div><h3>Results:</h3><p>The Word2vec model showed the highest consistency with the WASI-II manual (frequency = 20 out of 27; Pearson Correlation coefficient = 0.61) while Bidirectional Encoder Representations from Transformers was the worst performing model (frequency = 5; Pearson Correlation coefficient = 0.05). The consistency of these two models with the WASI-II manual differed significantly, Z = 2.282, p = 0.022.</p></div><div><h3>Conclusions:</h3><p>Our results showed that the scoring of the WASI-II Vocabulary task can be automated with moderate accuracy relying upon off-the-shelf embedding models. These results are promising, and could be improved further by considering alternative vector dimensions, similarity metrics, and data preprocessing techniques to those used in this study.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"8 ","pages":"Article 100094"},"PeriodicalIF":0.0,"publicationDate":"2024-08-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2949719124000426/pdfft?md5=3d77e8547a0dc7357280cecba9e28c62&pid=1-s2.0-S2949719124000426-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141952741","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Providing Citations to Support Fact-Checking: Contextualizing Detection of Sentences Needing Citation on Small Wikipedias","authors":"Aida Halitaj,&nbsp;Arkaitz Zubiaga","doi":"10.1016/j.nlp.2024.100093","DOIUrl":"10.1016/j.nlp.2024.100093","url":null,"abstract":"<div><p>Authoritative citations are critical to ensure information integrity, especially in encyclopedias like Wikipedia. To date, research on automating citation worthiness detection has largely focused on the most resourceful language, English Wikipedia, neglecting the applicability to smaller Wikipedias. In addition, previous research proposed models that analyze the content inherent to a sentence to determine its citation worthiness, overlooking the potential of additional context to improve the prediction. Addressing these gaps, our study proposes a transformer-based contextualized approach for smaller Wikipedias, presenting a novel method to compile high-quality datasets for the Albanian, Basque, and Catalan editions. We develop the <strong>C</strong>ontextualized <strong>C</strong>itation <strong>W</strong>orthiness (CCW) model, employing sentence representations enriched with adjacent sentences and topic categories for enhanced contextual insight. Empirical experiments on three newly created datasets demonstrate significant performance improvements of our contextualized CCW model, with 6%, 3% and 6% absolute improvements over the baseline for Albanian, Basque and Catalan datasets, respectively. We conduct an in-depth analysis to understand the influence and extent to which preceding and succeeding context as well as topic categories contribute to the accuracy of citation-worthiness predictions. Our findings suggest that incorporating such contextual information aids in the automatic identification of sentences in need of citations, not least when both the preceding and succeeding context are incorporated. This has implications for supporting Wikipedia projects across low-resource languages, promoting better article validation and fact-checking.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"8 ","pages":"Article 100093"},"PeriodicalIF":0.0,"publicationDate":"2024-08-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2949719124000414/pdfft?md5=5d5c2344f9651734d9e20fc37a799aae&pid=1-s2.0-S2949719124000414-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141992927","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Contrastive adversarial gender debiasing","authors":"Nicolás Torres","doi":"10.1016/j.nlp.2024.100092","DOIUrl":"10.1016/j.nlp.2024.100092","url":null,"abstract":"<div><p>This research contributes a comprehensive analysis of gender bias within contemporary AI language models, specifically examining iterations of the GPT series, alongside Gemini and Llama. The study offers a systematic investigation, encompassing multiple experiments spanning sentence completions, generative narratives, bilingual analysis, and visual perception assessments. The primary objective is to scrutinize the evolution of gender bias in these models across iterations, explore biases in professions and contexts, and evaluate multilingual disparities. Notably, the analyses reveal a marked evolution in GPT iterations, with GPT4 showcasing significantly reduced or negligible biases, signifying substantial advancements in bias mitigation. Professions and contexts exhibit model biases, indicating associations with specific genders. Multilingual evaluations demonstrate subtle disparities in gender bias tendencies between English and Spanish narratives. To effectively mitigate these biases, we propose a novel Contrastive Adversarial Gender Debiasing (CAGD) method that synergistically combines contrastive learning and adversarial training techniques. The CAGD method enables language models to learn gender-neutral representations while promoting robustness against gender biases, consistently outperforming original and adversarially debiased models across various tasks and metrics. These findings underscore the complexity of gender bias in AI language models, emphasizing the need for continual bias mitigation strategies, such as the proposed CAGD approach, and ethical considerations in AI development and deployment.</p></div>","PeriodicalId":100944,"journal":{"name":"Natural Language Processing Journal","volume":"8 ","pages":"Article 100092"},"PeriodicalIF":0.0,"publicationDate":"2024-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2949719124000402/pdfft?md5=8cd94a1ff5d32d6fad021f90862cb81a&pid=1-s2.0-S2949719124000402-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141848697","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}