Investigating the Effectiveness of COSMIC Method for Sizing Chatbots' Requirements: A Comparative Analysis

Abstract

In the modern digital era, chatbots have emerged as vital AI-powered systems owing to their usefulness and time savings, increasing companies' interest in developing or integrating these AI systems. Proper evaluation of chatbots requirements plays a critical role in managing their development projects and improving their quality and adoption. Although chatbots can be evaluated using the COSMIC FSM method, previous studies have mainly focused on intuitively evaluating the quality requirements of chatbots, often overlooking standardized assessments. In this paper, we report the results of applying COSMIC—ISO 19761 to chatbot requirements through the BotCFP tool. We selected 41 chatbots from web and mobile stores. The selected chatbots were ranked according to their relative COSMIC sizes and the scores obtained from a quality assessment questionnaire. We compared functional size rankings with functionality score rankings, then examined the relationship between chatbots' functional sizes and average users' ratings, alongside the correlation between mobile chatbot downloads and web chatbot visits. The results showed that 68.7% of the chatbots have similar ranking in both functional size and functionality score. Furthermore, there was no correlation between the functional size of the chatbots and their average users' ratings, mobile downloads, and web visits. The analysis of the results shows the effectiveness of the COSMIC FSM in assessing the chatbot requirements of the application layer. Our investigation highlights the need to extend COSMIC to support in-depth evaluation of chatbots and AI-based systems at the AI layer in future work.