Artificial Intelligence to Support Qualitative Data Analysis: Promises, Approaches, Pitfalls.

Abstract

Abstract: How can artificial intelligence (AI) be used to support qualitative data analysis (QDA)? To address this question, the authors conducted 3 scholarly activities. First, they used a readily available large language model, ChatGPT-4, to analyze 3 existing narrative datasets (February 2024). ChatGPT generated accurate brief summaries; for all other attempted tasks the initial prompt failed to produce desired results. After iterative prompt engineering, some tasks (e.g., keyword counting, summarization) were successful, whereas others (e.g., thematic analysis, keyword highlighting, word tree diagram, cross-theme insights) never generated satisfactory results. Second, the authors conducted a brief scoping review of AI-supported QDA (through May 2024). They identified 130 articles (104 original research, 26 nonresearch) of which 64 were published in 2023 or 2024. Seventy studies inductively analyzed data for themes, 39 used keyword detection, 30 applied a coding rubric, 28 used sentiment analysis, and 13 applied discourse analysis. Seventy-five used unsupervised learning (e.g., transformers, other neural networks). Third, building on these experiences and drawing from additional literature, the authors examined the potential capabilities, shortcomings, dangers, and ethical repercussions of AI-supported QDA. They note that AI has been used for QDA for more than 25 years. AI-supported QDA approaches include inductive and deductive coding, thematic analysis, computational grounded theory, discourse analysis, analysis of large datasets, preanalysis transcription and translation, and suggestions for study planning and interpretation. Concerns include the imperative of a "human in the loop" for data collection and analysis, the need for researchers to understand the technology, the risk of unsophisticated analyses, inevitable influences on workforce, and apprehensions regarding data privacy and security. Reflexivity should embrace both strengths and weaknesses of AI-supported QDA. The authors conclude that AI has a long history of supporting QDA through widely varied methods. Evolving technologies make AI-supported QDA more accessible and introduce both promises and pitfalls.