Advancing qualitative analysis in professional disaster and risk communication: A comparative study of an OpenAI ChatGPT 3.5 model-enabled method for processing complex public discourse
Margaret Webb , Harman Singh , Rachel Inman , Sweta Baniya , Andrew Katz
{"title":"Advancing qualitative analysis in professional disaster and risk communication: A comparative study of an OpenAI ChatGPT 3.5 model-enabled method for processing complex public discourse","authors":"Margaret Webb , Harman Singh , Rachel Inman , Sweta Baniya , Andrew Katz","doi":"10.1016/j.ijdrr.2025.105835","DOIUrl":null,"url":null,"abstract":"<div><div>Crisis managers and risk communicators face increasing challenges analyzing social media discourse during interconnected crises. This paper introduces an end-to-end method for qualitative codebook generation using the ChatGPT-3.5 Generative Artificial Intelligence (GenAI) Large Language Model (LLM) and Generative Pre-trained Transformer (GPT) from OpenAI, comparing a human-in-the-loop GenAI-enabled approach with traditional qualitative coding through a case study of Twitter discourse on COVID-19 and climate change in Virginia (2020–2022).</div><div>Methods build on prior work establishing qualitative analysis enhanced by Natural Language Processing (NLP) by stacking multiple iterative coding processes with GenAI and humans-in-the-loop to generate qualitative codebooks. A comparative analysis establishes recommendations for assessing the validity of the GPT-enabled generated codebooks. Human validation confirmed substantial concordance (91.7 % agreement) with the GPT-enabled process's coding, revealing structural similarities and distinct patterns between the two approaches' representation of analysis.</div><div>This research's contribution is methodological, establishing an approach for conducting and assessing GPT-enabled qualitative analysis while mapping relationships between computationally enhanced and traditional qualitative coding approaches. Findings advance risk communication methods by offering empirical guidance on integrating AI-assisted techniques within traditional qualitative research to maintain analytical rigor. Compared to prior NLP-enabled methods for disaster-related social media discourse analysis, this iterative approach mimics aspects of traditional qualitative codebook development, yielding multi-tiered codebook structures that can capture aspects of complex disaster discourse. While findings demonstrate that GenAI can enable analytical efficiency of traditional (human-only) codebook generation through self-assessment and iterative improvement with the help of humans-in-the-loop, they also illuminate areas of coding process where human expertise remains essential.</div></div>","PeriodicalId":13915,"journal":{"name":"International journal of disaster risk reduction","volume":"130 ","pages":"Article 105835"},"PeriodicalIF":4.5000,"publicationDate":"2025-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International journal of disaster risk reduction","FirstCategoryId":"89","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2212420925006594","RegionNum":1,"RegionCategory":"地球科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"GEOSCIENCES, MULTIDISCIPLINARY","Score":null,"Total":0}
引用次数: 0
Abstract
Crisis managers and risk communicators face increasing challenges analyzing social media discourse during interconnected crises. This paper introduces an end-to-end method for qualitative codebook generation using the ChatGPT-3.5 Generative Artificial Intelligence (GenAI) Large Language Model (LLM) and Generative Pre-trained Transformer (GPT) from OpenAI, comparing a human-in-the-loop GenAI-enabled approach with traditional qualitative coding through a case study of Twitter discourse on COVID-19 and climate change in Virginia (2020–2022).
Methods build on prior work establishing qualitative analysis enhanced by Natural Language Processing (NLP) by stacking multiple iterative coding processes with GenAI and humans-in-the-loop to generate qualitative codebooks. A comparative analysis establishes recommendations for assessing the validity of the GPT-enabled generated codebooks. Human validation confirmed substantial concordance (91.7 % agreement) with the GPT-enabled process's coding, revealing structural similarities and distinct patterns between the two approaches' representation of analysis.
This research's contribution is methodological, establishing an approach for conducting and assessing GPT-enabled qualitative analysis while mapping relationships between computationally enhanced and traditional qualitative coding approaches. Findings advance risk communication methods by offering empirical guidance on integrating AI-assisted techniques within traditional qualitative research to maintain analytical rigor. Compared to prior NLP-enabled methods for disaster-related social media discourse analysis, this iterative approach mimics aspects of traditional qualitative codebook development, yielding multi-tiered codebook structures that can capture aspects of complex disaster discourse. While findings demonstrate that GenAI can enable analytical efficiency of traditional (human-only) codebook generation through self-assessment and iterative improvement with the help of humans-in-the-loop, they also illuminate areas of coding process where human expertise remains essential.
期刊介绍:
The International Journal of Disaster Risk Reduction (IJDRR) is the journal for researchers, policymakers and practitioners across diverse disciplines: earth sciences and their implications; environmental sciences; engineering; urban studies; geography; and the social sciences. IJDRR publishes fundamental and applied research, critical reviews, policy papers and case studies with a particular focus on multi-disciplinary research that aims to reduce the impact of natural, technological, social and intentional disasters. IJDRR stimulates exchange of ideas and knowledge transfer on disaster research, mitigation, adaptation, prevention and risk reduction at all geographical scales: local, national and international.
Key topics:-
-multifaceted disaster and cascading disasters
-the development of disaster risk reduction strategies and techniques
-discussion and development of effective warning and educational systems for risk management at all levels
-disasters associated with climate change
-vulnerability analysis and vulnerability trends
-emerging risks
-resilience against disasters.
The journal particularly encourages papers that approach risk from a multi-disciplinary perspective.