Comprehensive analysis of Transformer networks in identifying informative sentences containing customer needs

The unprecedented rise in user-generated content (UGC) provides businesses with new opportunities to extract customer insights from unstructured data, particularly for identifying customer needs. Intelligent methods offer time- and cost-efficient solutions to extract such insights from the plethora of repetitive and often redundant UGC. However, widespread adoption of these methods faces significant barriers, including high deployment and maintenance costs, data availability challenges, task complexity, and concerns about model efficacy and ethical implications. To facilitate broader adoption of intelligent systems in customer needs analysis, this study evaluates Transformer-based models in terms of generalizability, fairness, robustness, and sample efficiency across various experimental settings to uncover their true performance and identify the root causes of their errors. Our results show that although Transformer-based models improved the F1-score by up to 18% compared to baselines, their limitations become evident when evaluating their performance against task objectives. Key findings include: (i) Transformer-based networks share error patterns and struggle to identify infrequent or unseen informative samples, (ii) they heavily rely on abundant information and lexical cues for accurate predictions, compromising inter- and intra-domain generalizability, (iii) larger models do not necessarily improve sample efficiency within their domain, and (iv) while optimal cross-domain results arise from complex domain training, adding more in-domain samples does not enhance cross-domain performance. Overall, this research provides crucial insights to help businesses overcome adoption barriers when implementing Natural Language Processing advancements, such as Transformer-based models, in the customer needs analysis process. Source codes are available at https://github.com/mehrshad-kashi/ISCN-UsingTransformerNetworks.