Mehrshad Kashi , Salim Lahmiri , Otmane Ait Mohamed
{"title":"Comprehensive analysis of Transformer networks in identifying informative sentences containing customer needs","authors":"Mehrshad Kashi , Salim Lahmiri , Otmane Ait Mohamed","doi":"10.1016/j.eswa.2025.126785","DOIUrl":null,"url":null,"abstract":"<div><div>The unprecedented rise in user-generated content (UGC) provides businesses with new opportunities to extract customer insights from unstructured data, particularly for identifying customer needs. Intelligent methods offer time- and cost-efficient solutions to extract such insights from the plethora of repetitive and often redundant UGC. However, widespread adoption of these methods faces significant barriers, including high deployment and maintenance costs, data availability challenges, task complexity, and concerns about model efficacy and ethical implications. To facilitate broader adoption of intelligent systems in customer needs analysis, this study evaluates Transformer-based models in terms of generalizability, fairness, robustness, and sample efficiency across various experimental settings to uncover their true performance and identify the root causes of their errors. Our results show that although Transformer-based models improved the F1-score by up to 18% compared to baselines, their limitations become evident when evaluating their performance against task objectives. Key findings include: (i) Transformer-based networks share error patterns and struggle to identify infrequent or unseen informative samples, (ii) they heavily rely on abundant information and lexical cues for accurate predictions, compromising inter- and intra-domain generalizability, (iii) larger models do not necessarily improve sample efficiency within their domain, and (iv) while optimal cross-domain results arise from complex domain training, adding more in-domain samples does not enhance cross-domain performance. Overall, this research provides crucial insights to help businesses overcome adoption barriers when implementing Natural Language Processing advancements, such as Transformer-based models, in the customer needs analysis process. Source codes are available at <span><span>https://github.com/mehrshad-kashi/ISCN-UsingTransformerNetworks</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":50461,"journal":{"name":"Expert Systems with Applications","volume":"273 ","pages":"Article 126785"},"PeriodicalIF":7.5000,"publicationDate":"2025-02-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Expert Systems with Applications","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0957417425004075","RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
Abstract
The unprecedented rise in user-generated content (UGC) provides businesses with new opportunities to extract customer insights from unstructured data, particularly for identifying customer needs. Intelligent methods offer time- and cost-efficient solutions to extract such insights from the plethora of repetitive and often redundant UGC. However, widespread adoption of these methods faces significant barriers, including high deployment and maintenance costs, data availability challenges, task complexity, and concerns about model efficacy and ethical implications. To facilitate broader adoption of intelligent systems in customer needs analysis, this study evaluates Transformer-based models in terms of generalizability, fairness, robustness, and sample efficiency across various experimental settings to uncover their true performance and identify the root causes of their errors. Our results show that although Transformer-based models improved the F1-score by up to 18% compared to baselines, their limitations become evident when evaluating their performance against task objectives. Key findings include: (i) Transformer-based networks share error patterns and struggle to identify infrequent or unseen informative samples, (ii) they heavily rely on abundant information and lexical cues for accurate predictions, compromising inter- and intra-domain generalizability, (iii) larger models do not necessarily improve sample efficiency within their domain, and (iv) while optimal cross-domain results arise from complex domain training, adding more in-domain samples does not enhance cross-domain performance. Overall, this research provides crucial insights to help businesses overcome adoption barriers when implementing Natural Language Processing advancements, such as Transformer-based models, in the customer needs analysis process. Source codes are available at https://github.com/mehrshad-kashi/ISCN-UsingTransformerNetworks.
期刊介绍:
Expert Systems With Applications is an international journal dedicated to the exchange of information on expert and intelligent systems used globally in industry, government, and universities. The journal emphasizes original papers covering the design, development, testing, implementation, and management of these systems, offering practical guidelines. It spans various sectors such as finance, engineering, marketing, law, project management, information management, medicine, and more. The journal also welcomes papers on multi-agent systems, knowledge management, neural networks, knowledge discovery, data mining, and other related areas, excluding applications to military/defense systems.