Exploring the Use of a Large Language Model for Inductive Content Analysis in a Discourse Network Analysis Study

IF 3 2区社会学 Q2 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS

Social Science Computer Review Pub Date : 2025-03-14 DOI:10.1177/08944393251326175

Steve Randerson, Thomas Graydon-Guy, En-Yi Lin, Sally Casswell

{"title":"Exploring the Use of a Large Language Model for Inductive Content Analysis in a Discourse Network Analysis Study","authors":"Steve Randerson, Thomas Graydon-Guy, En-Yi Lin, Sally Casswell","doi":"10.1177/08944393251326175","DOIUrl":null,"url":null,"abstract":"Large language models show promising capability in some qualitative content analysis tasks; however, research reporting their performance in identifying initial codes that underpin subsequent analysis is scarce. This paper explores the suitability of GPT-4 to assist in building a codebook for a discourse network analysis (DNA) of a recent alcohol policy reform. DNA is a codebook-driven approach to identifying groupings of actors who use similar policy framings. The paper uses GPT-4 to identify initial codes (‘concepts’) and related quotes in 108 news articles and interviews. The results produced by GPT-4 are compared to a codebook prepared by researchers. GPT-4 identified over two-thirds of the concepts found by the researchers, and it was highly accurate in screening out a large volume of irrelevant media items. However, GPT-4 also provided many irrelevant concepts that required researcher review and removal. The discussion reflects on the implications for using GPT-4 in codebook preparation for DNA and other situations, including the need for human involvement and sample testing to understand its strengths and limitations, which may limit efficiency gains.","PeriodicalId":49509,"journal":{"name":"Social Science Computer Review","volume":"56 1","pages":""},"PeriodicalIF":3.0000,"publicationDate":"2025-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Social Science Computer Review","FirstCategoryId":"90","ListUrlMain":"https://doi.org/10.1177/08944393251326175","RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}

引用次数: 0

Abstract

Large language models show promising capability in some qualitative content analysis tasks; however, research reporting their performance in identifying initial codes that underpin subsequent analysis is scarce. This paper explores the suitability of GPT-4 to assist in building a codebook for a discourse network analysis (DNA) of a recent alcohol policy reform. DNA is a codebook-driven approach to identifying groupings of actors who use similar policy framings. The paper uses GPT-4 to identify initial codes (‘concepts’) and related quotes in 108 news articles and interviews. The results produced by GPT-4 are compared to a codebook prepared by researchers. GPT-4 identified over two-thirds of the concepts found by the researchers, and it was highly accurate in screening out a large volume of irrelevant media items. However, GPT-4 also provided many irrelevant concepts that required researcher review and removal. The discussion reflects on the implications for using GPT-4 in codebook preparation for DNA and other situations, including the need for human involvement and sample testing to understand its strengths and limitations, which may limit efficiency gains.

查看原文本刊更多论文

大语言模型在语篇网络分析研究中的应用探讨

大型语言模型在一些定性内容分析任务中显示出良好的能力；然而，报告它们在识别支撑后续分析的初始代码方面的表现的研究很少。本文探讨了GPT-4的适用性，以协助建立一个密码本的话语网络分析（DNA）最近的酒精政策改革。DNA是一种代码本驱动的方法，用于识别使用类似政策框架的行为者群体。本文使用GPT-4识别108篇新闻文章和采访中的初始代码（“概念”）和相关引用。GPT-4产生的结果与研究人员准备的密码本进行了比较。GPT-4识别了研究人员发现的超过三分之二的概念，并且在筛选大量不相关的媒体项目方面非常准确。然而，GPT-4也提供了许多不相关的概念，需要研究人员审查和删除。讨论反映了在DNA和其他情况的码本制备中使用GPT-4的影响，包括需要人工参与和样本测试以了解其优势和局限性，这可能会限制效率的提高。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Social Science Computer Review 社会科学-计算机：跨学科应用

CiteScore

9.00

自引率

4.90%

发文量

审稿时长

>12 weeks

期刊介绍： Unique Scope Social Science Computer Review is an interdisciplinary journal covering social science instructional and research applications of computing, as well as societal impacts of informational technology. Topics included: artificial intelligence, business, computational social science theory, computer-assisted survey research, computer-based qualitative analysis, computer simulation, economic modeling, electronic modeling, electronic publishing, geographic information systems, instrumentation and research tools, public administration, social impacts of computing and telecommunications, software evaluation, world-wide web resources for social scientists. Interdisciplinary Nature Because the Uses and impacts of computing are interdisciplinary, so is Social Science Computer Review. The journal is of direct relevance to scholars and scientists in a wide variety of disciplines. In its pages you''ll find work in the following areas: sociology, anthropology, political science, economics, psychology, computer literacy, computer applications, and methodology.