ADVISE:ADVISE:利用自然语言处理支持的人与人工智能协作,加速创建全球发展证据综合体

IF 2.9 3区 工程技术 Q2 ENGINEERING, MECHANICAL
Kristen M. Edwards, Binyang Song, Jaron Porciello, Mark Engelbert, Carolyn Huang, Faez Ahmed
{"title":"ADVISE:ADVISE:利用自然语言处理支持的人与人工智能协作,加速创建全球发展证据综合体","authors":"Kristen M. Edwards, Binyang Song, Jaron Porciello, Mark Engelbert, Carolyn Huang, Faez Ahmed","doi":"10.1115/1.4064245","DOIUrl":null,"url":null,"abstract":"\n When designing evidence-based policies and programs, decision-makers must distill key information from a vast and rapidly growing literature base. Identifying relevant literature from raw search results is time and resource intensive, and is often done by manual screening. In this study, we develop an AI agent based on a bidirectional encoder representations from transformers (BERT) model and incorporate it into a human team designing an evidence synthesis product for global development. We explore the effectiveness of the human-AI hybrid team in accelerating the evidence synthesis process. We further enhance the human-AI hybrid team through active learning (AL). Specifically, we explore different sampling strategies: random, least confidence (LC), and highest priority (HP) sampling, to study their influence on the collaborative screening process. Results show that incorporating the BERT-based AI agent can reduce the human screening effort by 68.5% compared to the case of no AI assistance, and by 16.8% compared to using the industry standard model for identifying 80% of all relevant documents. When we apply the HP sampling strategy, the human screening effort can be reduced even more: by 78% for identifying 80% of all relevant documents compared to no AI assistance. We apply the AL-enhanced human-AI hybrid teaming workflow in the design process of three evidence gap maps which are now published for USAID's use. These findings demonstrate how AI can accelerate the development of evidence synthesis products and promote timely evidence-based decision making in global development.","PeriodicalId":50137,"journal":{"name":"Journal of Mechanical Design","volume":null,"pages":null},"PeriodicalIF":2.9000,"publicationDate":"2023-12-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"ADVISE: Accelerating the Creation of Evidence Syntheses for Global Development using Natural Language Processing-supported Human-AI Collaboration\",\"authors\":\"Kristen M. Edwards, Binyang Song, Jaron Porciello, Mark Engelbert, Carolyn Huang, Faez Ahmed\",\"doi\":\"10.1115/1.4064245\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"\\n When designing evidence-based policies and programs, decision-makers must distill key information from a vast and rapidly growing literature base. Identifying relevant literature from raw search results is time and resource intensive, and is often done by manual screening. In this study, we develop an AI agent based on a bidirectional encoder representations from transformers (BERT) model and incorporate it into a human team designing an evidence synthesis product for global development. We explore the effectiveness of the human-AI hybrid team in accelerating the evidence synthesis process. We further enhance the human-AI hybrid team through active learning (AL). Specifically, we explore different sampling strategies: random, least confidence (LC), and highest priority (HP) sampling, to study their influence on the collaborative screening process. Results show that incorporating the BERT-based AI agent can reduce the human screening effort by 68.5% compared to the case of no AI assistance, and by 16.8% compared to using the industry standard model for identifying 80% of all relevant documents. When we apply the HP sampling strategy, the human screening effort can be reduced even more: by 78% for identifying 80% of all relevant documents compared to no AI assistance. We apply the AL-enhanced human-AI hybrid teaming workflow in the design process of three evidence gap maps which are now published for USAID's use. These findings demonstrate how AI can accelerate the development of evidence synthesis products and promote timely evidence-based decision making in global development.\",\"PeriodicalId\":50137,\"journal\":{\"name\":\"Journal of Mechanical Design\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":2.9000,\"publicationDate\":\"2023-12-07\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Mechanical Design\",\"FirstCategoryId\":\"5\",\"ListUrlMain\":\"https://doi.org/10.1115/1.4064245\",\"RegionNum\":3,\"RegionCategory\":\"工程技术\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"ENGINEERING, MECHANICAL\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Mechanical Design","FirstCategoryId":"5","ListUrlMain":"https://doi.org/10.1115/1.4064245","RegionNum":3,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"ENGINEERING, MECHANICAL","Score":null,"Total":0}
引用次数: 0

摘要

在设计基于证据的政策和项目时,决策者必须从庞大且快速增长的文献基础中提取关键信息。从原始搜索结果中识别相关文献需要大量的时间和资源,并且通常是通过人工筛选完成的。在这项研究中,我们开发了一个基于双向编码器表示(BERT)模型的人工智能代理,并将其纳入一个为全球开发设计证据合成产品的人类团队。我们探讨了人工智能混合团队在加速证据合成过程中的有效性。我们通过主动学习(AL)进一步加强人类-人工智能混合团队。具体而言,我们探讨了不同的抽样策略:随机、最小置信度(LC)和最高优先级(HP)抽样,以研究它们对协同筛选过程的影响。结果表明,与没有人工智能辅助的情况相比,结合基于bert的人工智能代理可以减少68.5%的人工筛选工作量,与使用行业标准模型识别80%的所有相关文档相比,可以减少16.8%的人工筛选工作量。当我们应用惠普抽样策略时,人工筛选工作可以减少更多:与没有人工智能帮助相比,识别80%的所有相关文件可减少78%。我们将人工智能增强的人类-人工智能混合团队工作流程应用于三个证据差距地图的设计过程中,这些地图现已发布,供美国国际开发署使用。这些发现表明,人工智能可以加速证据合成产品的开发,并在全球发展中促进及时的循证决策。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
ADVISE: Accelerating the Creation of Evidence Syntheses for Global Development using Natural Language Processing-supported Human-AI Collaboration
When designing evidence-based policies and programs, decision-makers must distill key information from a vast and rapidly growing literature base. Identifying relevant literature from raw search results is time and resource intensive, and is often done by manual screening. In this study, we develop an AI agent based on a bidirectional encoder representations from transformers (BERT) model and incorporate it into a human team designing an evidence synthesis product for global development. We explore the effectiveness of the human-AI hybrid team in accelerating the evidence synthesis process. We further enhance the human-AI hybrid team through active learning (AL). Specifically, we explore different sampling strategies: random, least confidence (LC), and highest priority (HP) sampling, to study their influence on the collaborative screening process. Results show that incorporating the BERT-based AI agent can reduce the human screening effort by 68.5% compared to the case of no AI assistance, and by 16.8% compared to using the industry standard model for identifying 80% of all relevant documents. When we apply the HP sampling strategy, the human screening effort can be reduced even more: by 78% for identifying 80% of all relevant documents compared to no AI assistance. We apply the AL-enhanced human-AI hybrid teaming workflow in the design process of three evidence gap maps which are now published for USAID's use. These findings demonstrate how AI can accelerate the development of evidence synthesis products and promote timely evidence-based decision making in global development.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Journal of Mechanical Design
Journal of Mechanical Design 工程技术-工程:机械
CiteScore
8.00
自引率
18.20%
发文量
139
审稿时长
3.9 months
期刊介绍: The Journal of Mechanical Design (JMD) serves the broad design community as the venue for scholarly, archival research in all aspects of the design activity with emphasis on design synthesis. JMD has traditionally served the ASME Design Engineering Division and its technical committees, but it welcomes contributions from all areas of design with emphasis on synthesis. JMD communicates original contributions, primarily in the form of research articles of considerable depth, but also technical briefs, design innovation papers, book reviews, and editorials. Scope: The Journal of Mechanical Design (JMD) serves the broad design community as the venue for scholarly, archival research in all aspects of the design activity with emphasis on design synthesis. JMD has traditionally served the ASME Design Engineering Division and its technical committees, but it welcomes contributions from all areas of design with emphasis on synthesis. JMD communicates original contributions, primarily in the form of research articles of considerable depth, but also technical briefs, design innovation papers, book reviews, and editorials.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信