Advancing qualitative analysis in professional disaster and risk communication: A comparative study of an OpenAI ChatGPT 3.5 model-enabled method for processing complex public discourse

IF 4.5 1区 地球科学 Q1 GEOSCIENCES, MULTIDISCIPLINARY
Margaret Webb , Harman Singh , Rachel Inman , Sweta Baniya , Andrew Katz
{"title":"Advancing qualitative analysis in professional disaster and risk communication: A comparative study of an OpenAI ChatGPT 3.5 model-enabled method for processing complex public discourse","authors":"Margaret Webb ,&nbsp;Harman Singh ,&nbsp;Rachel Inman ,&nbsp;Sweta Baniya ,&nbsp;Andrew Katz","doi":"10.1016/j.ijdrr.2025.105835","DOIUrl":null,"url":null,"abstract":"<div><div>Crisis managers and risk communicators face increasing challenges analyzing social media discourse during interconnected crises. This paper introduces an end-to-end method for qualitative codebook generation using the ChatGPT-3.5 Generative Artificial Intelligence (GenAI) Large Language Model (LLM) and Generative Pre-trained Transformer (GPT) from OpenAI, comparing a human-in-the-loop GenAI-enabled approach with traditional qualitative coding through a case study of Twitter discourse on COVID-19 and climate change in Virginia (2020–2022).</div><div>Methods build on prior work establishing qualitative analysis enhanced by Natural Language Processing (NLP) by stacking multiple iterative coding processes with GenAI and humans-in-the-loop to generate qualitative codebooks. A comparative analysis establishes recommendations for assessing the validity of the GPT-enabled generated codebooks. Human validation confirmed substantial concordance (91.7 % agreement) with the GPT-enabled process's coding, revealing structural similarities and distinct patterns between the two approaches' representation of analysis.</div><div>This research's contribution is methodological, establishing an approach for conducting and assessing GPT-enabled qualitative analysis while mapping relationships between computationally enhanced and traditional qualitative coding approaches. Findings advance risk communication methods by offering empirical guidance on integrating AI-assisted techniques within traditional qualitative research to maintain analytical rigor. Compared to prior NLP-enabled methods for disaster-related social media discourse analysis, this iterative approach mimics aspects of traditional qualitative codebook development, yielding multi-tiered codebook structures that can capture aspects of complex disaster discourse. While findings demonstrate that GenAI can enable analytical efficiency of traditional (human-only) codebook generation through self-assessment and iterative improvement with the help of humans-in-the-loop, they also illuminate areas of coding process where human expertise remains essential.</div></div>","PeriodicalId":13915,"journal":{"name":"International journal of disaster risk reduction","volume":"130 ","pages":"Article 105835"},"PeriodicalIF":4.5000,"publicationDate":"2025-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International journal of disaster risk reduction","FirstCategoryId":"89","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2212420925006594","RegionNum":1,"RegionCategory":"地球科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"GEOSCIENCES, MULTIDISCIPLINARY","Score":null,"Total":0}
引用次数: 0

Abstract

Crisis managers and risk communicators face increasing challenges analyzing social media discourse during interconnected crises. This paper introduces an end-to-end method for qualitative codebook generation using the ChatGPT-3.5 Generative Artificial Intelligence (GenAI) Large Language Model (LLM) and Generative Pre-trained Transformer (GPT) from OpenAI, comparing a human-in-the-loop GenAI-enabled approach with traditional qualitative coding through a case study of Twitter discourse on COVID-19 and climate change in Virginia (2020–2022).
Methods build on prior work establishing qualitative analysis enhanced by Natural Language Processing (NLP) by stacking multiple iterative coding processes with GenAI and humans-in-the-loop to generate qualitative codebooks. A comparative analysis establishes recommendations for assessing the validity of the GPT-enabled generated codebooks. Human validation confirmed substantial concordance (91.7 % agreement) with the GPT-enabled process's coding, revealing structural similarities and distinct patterns between the two approaches' representation of analysis.
This research's contribution is methodological, establishing an approach for conducting and assessing GPT-enabled qualitative analysis while mapping relationships between computationally enhanced and traditional qualitative coding approaches. Findings advance risk communication methods by offering empirical guidance on integrating AI-assisted techniques within traditional qualitative research to maintain analytical rigor. Compared to prior NLP-enabled methods for disaster-related social media discourse analysis, this iterative approach mimics aspects of traditional qualitative codebook development, yielding multi-tiered codebook structures that can capture aspects of complex disaster discourse. While findings demonstrate that GenAI can enable analytical efficiency of traditional (human-only) codebook generation through self-assessment and iterative improvement with the help of humans-in-the-loop, they also illuminate areas of coding process where human expertise remains essential.
推进专业灾害和风险沟通中的定性分析:OpenAI ChatGPT 3.5模型支持的复杂公共话语处理方法的比较研究
危机管理者和风险传播者在相互关联的危机中分析社交媒体话语面临越来越大的挑战。本文介绍了一种使用OpenAI的ChatGPT-3.5生成式人工智能(GenAI)大型语言模型(LLM)和生成式预训练转换器(GPT)进行定性代码本生成的端到端方法,并通过对Twitter上关于2019冠状病毒病和弗吉尼亚州气候变化的讨论(2020-2022)的案例研究,将基于人在环的GenAI方法与传统定性编码进行了比较。方法建立在自然语言处理(NLP)增强定性分析的基础上,通过将多个迭代编码过程与GenAI和human -in-the-loop叠加在一起,生成定性代码本。比较分析为评估启用gpt的生成码本的有效性建立了建议。人类验证证实了与gpt支持的过程编码的实质性一致性(91.7%的一致性),揭示了两种方法分析表示之间的结构相似性和不同模式。本研究的贡献是方法论,建立了一种方法来进行和评估gpt支持的定性分析,同时映射计算增强和传统定性编码方法之间的关系。研究结果通过提供在传统定性研究中整合人工智能辅助技术以保持分析严谨性的经验指导,推进了风险沟通方法。与之前支持nlp的灾害相关社交媒体话语分析方法相比,这种迭代方法模仿了传统定性代码本开发的各个方面,产生了多层代码本结构,可以捕获复杂灾难话语的各个方面。虽然研究结果表明,GenAI可以通过人类在循环中的自我评估和迭代改进来提高传统(只有人类)代码本生成的分析效率,但它们也阐明了编码过程中人类专业知识仍然必不可少的领域。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
International journal of disaster risk reduction
International journal of disaster risk reduction GEOSCIENCES, MULTIDISCIPLINARYMETEOROLOGY-METEOROLOGY & ATMOSPHERIC SCIENCES
CiteScore
8.70
自引率
18.00%
发文量
688
审稿时长
79 days
期刊介绍: The International Journal of Disaster Risk Reduction (IJDRR) is the journal for researchers, policymakers and practitioners across diverse disciplines: earth sciences and their implications; environmental sciences; engineering; urban studies; geography; and the social sciences. IJDRR publishes fundamental and applied research, critical reviews, policy papers and case studies with a particular focus on multi-disciplinary research that aims to reduce the impact of natural, technological, social and intentional disasters. IJDRR stimulates exchange of ideas and knowledge transfer on disaster research, mitigation, adaptation, prevention and risk reduction at all geographical scales: local, national and international. Key topics:- -multifaceted disaster and cascading disasters -the development of disaster risk reduction strategies and techniques -discussion and development of effective warning and educational systems for risk management at all levels -disasters associated with climate change -vulnerability analysis and vulnerability trends -emerging risks -resilience against disasters. The journal particularly encourages papers that approach risk from a multi-disciplinary perspective.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信