From self-service to AI-assisted service: A mixed-method study of IT support service provision using search tools and chatbots

Abstract

Organizations are increasingly adopting chatbots for self-service IT support alongside traditional search tools to supplement human-staffed IT support desks. This raises the question of whether investing in chatbots is warranted, given the presence of well-established traditional search tools serving a similar function. We investigate how chatbots compare to self-service search tools from the user’s perspective. We conducted three experiments and a transcript analysis study in the context of self-service IT support using a chatbot versus a search tool, both running on the same large IT support knowledge base. All three experiments found higher user satisfaction with the chatbot than with the search tool. The main study investigated two potential theoretical mechanisms underlying these effects: (i) perceived assistance and (ii) co-creating questions with the chatbot to find an answer. Both significantly contributed to users’ satisfaction and willingness to use, with perceived assistance fully mediating the effect of the chatbot. The transcript analysis study showed that 57 % of user questions were quickly answered (1–2 conversation turns), and another 22 % quickly abandoned without an answer. Co-creation was important in successfully answering the remaining 21 % of questions that took longer than two conversation turns. We conclude that organizations can enhance their self-service IT support by integrating chatbots alongside existing traditional search tools. This transformation effectively shifts the self-service experience into an AI-assisted service experience for users.