Evaluating generative AI for qualitative data extraction in community-based fisheries management literature.

IF 3.4 4区 环境科学与生态学 Q2 ENVIRONMENTAL SCIENCES
S Spillias, K M Ollerhead, M Andreotta, R Annand-Jones, F Boschetti, J Duggan, D B Karcher, C Paris, R J Shellock, R Trebilco
{"title":"Evaluating generative AI for qualitative data extraction in community-based fisheries management literature.","authors":"S Spillias, K M Ollerhead, M Andreotta, R Annand-Jones, F Boschetti, J Duggan, D B Karcher, C Paris, R J Shellock, R Trebilco","doi":"10.1186/s13750-025-00362-9","DOIUrl":null,"url":null,"abstract":"<p><p>Uptake of AI tools in knowledge production processes is rapidly growing. In this pilot study, we explore the ability of generative AI tools to reliably extract qualitative data from a limited sample of peer-reviewed documents in the context of community-based fisheries management (CBFM) literature. Specifically, we evaluate the capacity of multiple AI tools to analyse 33 CBFM papers and extract relevant information for a systematic literature review, comparing the results to those of human reviewers. We address how well AI tools can discern the presence of relevant contextual data, whether the outputs of AI tools are comparable to human extractions, and whether the difficulty of question influences the performance of the extraction. While the AI tools we tested (GPT4-Turbo and Elicit) were not reliable in discerning the presence or absence of contextual data, at least one of the AI tools consistently returned responses that were on par with human reviewers. These results highlight the potential utility of AI tools in the extraction phase of evidence synthesis for supporting human-led reviews, while underscoring the ongoing need for human oversight. This exploratory investigation provides initial insights into the current capabilities and limitations of AI in qualitative data extraction within the specific domain of CBFM, laying groundwork for future, more comprehensive evaluations across diverse fields and larger datasets.</p>","PeriodicalId":48621,"journal":{"name":"Environmental Evidence","volume":"14 1","pages":"9"},"PeriodicalIF":3.4000,"publicationDate":"2025-06-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12128520/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Environmental Evidence","FirstCategoryId":"93","ListUrlMain":"https://doi.org/10.1186/s13750-025-00362-9","RegionNum":4,"RegionCategory":"环境科学与生态学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"ENVIRONMENTAL SCIENCES","Score":null,"Total":0}
引用次数: 0

Abstract

Uptake of AI tools in knowledge production processes is rapidly growing. In this pilot study, we explore the ability of generative AI tools to reliably extract qualitative data from a limited sample of peer-reviewed documents in the context of community-based fisheries management (CBFM) literature. Specifically, we evaluate the capacity of multiple AI tools to analyse 33 CBFM papers and extract relevant information for a systematic literature review, comparing the results to those of human reviewers. We address how well AI tools can discern the presence of relevant contextual data, whether the outputs of AI tools are comparable to human extractions, and whether the difficulty of question influences the performance of the extraction. While the AI tools we tested (GPT4-Turbo and Elicit) were not reliable in discerning the presence or absence of contextual data, at least one of the AI tools consistently returned responses that were on par with human reviewers. These results highlight the potential utility of AI tools in the extraction phase of evidence synthesis for supporting human-led reviews, while underscoring the ongoing need for human oversight. This exploratory investigation provides initial insights into the current capabilities and limitations of AI in qualitative data extraction within the specific domain of CBFM, laying groundwork for future, more comprehensive evaluations across diverse fields and larger datasets.

评估基于社区的渔业管理文献中定性数据提取的生成人工智能。
人工智能工具在知识生产过程中的应用正在迅速增长。在这项试点研究中,我们探索了生成式人工智能工具在社区渔业管理(CBFM)文献背景下,从有限的同行评审文件样本中可靠地提取定性数据的能力。具体来说,我们评估了多个人工智能工具分析33篇CBFM论文并提取相关信息以进行系统文献综述的能力,并将结果与人类审稿人的结果进行了比较。我们讨论了人工智能工具如何识别相关上下文数据的存在,人工智能工具的输出是否与人类提取相媲美,以及问题的难度是否会影响提取的性能。虽然我们测试的人工智能工具(GPT4-Turbo和Elicit)在识别上下文数据的存在与否方面并不可靠,但至少有一个人工智能工具始终返回与人类评论者相当的响应。这些结果突出了人工智能工具在证据合成的提取阶段的潜在效用,以支持人类主导的审查,同时强调了对人类监督的持续需求。这项探索性调查提供了对人工智能在CBFM特定领域定性数据提取方面的当前能力和局限性的初步见解,为未来在不同领域和更大数据集上进行更全面的评估奠定了基础。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Environmental Evidence
Environmental Evidence Environmental Science-Management, Monitoring, Policy and Law
CiteScore
6.10
自引率
18.20%
发文量
36
审稿时长
17 weeks
期刊介绍: Environmental Evidence is the journal of the Collaboration for Environmental Evidence (CEE). The Journal facilitates rapid publication of evidence syntheses, in the form of Systematic Reviews and Maps conducted to CEE Guidelines and Standards. We focus on the effectiveness of environmental management interventions and the impact of human activities on the environment. Our scope covers all forms of environmental management and human impacts and therefore spans the natural and social sciences. Subjects include water security, agriculture, food security, forestry, fisheries, natural resource management, biodiversity conservation, climate change, ecosystem services, pollution, invasive species, environment and human wellbeing, sustainable energy use, soil management, environmental legislation, environmental education.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信