Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies

Oscar Lau, Su Golder
{"title":"Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies","authors":"Oscar Lau,&nbsp;Su Golder","doi":"10.1002/cesm.70050","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>Elicit AI aims to simplify and accelerate the systematic review process without compromising accuracy. However, research on Elicit's performance is limited.</p>\n </section>\n \n <section>\n \n <h3> Objectives</h3>\n \n <p>To determine whether Elicit AI is a viable tool for systematic literature searches and title/abstract screening stages.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>We compared the included studies in four evidence syntheses to those identified using the subscription-based version of Elicit Pro in Review mode. We calculated sensitivity, precision and observed patterns in the performance of Elicit.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>The sensitivity of Elicit was poor, averaging 39.5% (25.5–69.2%) compared to 94.5% (91.1–98.0%) in the original reviews. However, Elicit identified some included studies not identified by the original searches and had an average of 41.8% precision (35.6–46.2%) which was higher than the 7.55% average of the original reviews (0.65–14.7%).</p>\n </section>\n \n <section>\n \n <h3> Discussion</h3>\n \n <p>At the time of this evaluation, Elicit did not search with high enough sensitivity to replace traditional literature searching. However, the high precision of searching in Elicit could prove useful for preliminary searches, and the unique studies identified mean that Elicit can be used by researchers as a useful adjunct.</p>\n </section>\n \n <section>\n \n <h3> Conclusion</h3>\n \n <p>Whilst Elicit searches are currently not sensitive enough to replace traditional searching, Elicit is continually improving, and further evaluations should be undertaken as new developments take place.</p>\n </section>\n </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2025-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70050","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Cochrane Evidence Synthesis and Methods","FirstCategoryId":"1085","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/cesm.70050","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Background

Elicit AI aims to simplify and accelerate the systematic review process without compromising accuracy. However, research on Elicit's performance is limited.

Objectives

To determine whether Elicit AI is a viable tool for systematic literature searches and title/abstract screening stages.

Methods

We compared the included studies in four evidence syntheses to those identified using the subscription-based version of Elicit Pro in Review mode. We calculated sensitivity, precision and observed patterns in the performance of Elicit.

Results

The sensitivity of Elicit was poor, averaging 39.5% (25.5–69.2%) compared to 94.5% (91.1–98.0%) in the original reviews. However, Elicit identified some included studies not identified by the original searches and had an average of 41.8% precision (35.6–46.2%) which was higher than the 7.55% average of the original reviews (0.65–14.7%).

Discussion

At the time of this evaluation, Elicit did not search with high enough sensitivity to replace traditional literature searching. However, the high precision of searching in Elicit could prove useful for preliminary searches, and the unique studies identified mean that Elicit can be used by researchers as a useful adjunct.

Conclusion

Whilst Elicit searches are currently not sensitive enough to replace traditional searching, Elicit is continually improving, and further evaluations should be undertaken as new developments take place.

Abstract Image

引导性人工智能与传统文献检索在证据综合中的比较——以四个案例为例
引出人工智能旨在简化和加速系统审查过程,而不影响准确性。然而,对Elicit的表现的研究是有限的。目的确定Elicit AI是否为系统文献检索和标题/摘要筛选阶段的可行工具。方法我们将四项证据综合纳入的研究与使用基于订阅的Elicit Pro在Review模式下识别的研究进行比较。我们计算了Elicit的灵敏度、精度和观察模式。结果Elicit的敏感性较差,平均为39.5%(25.5 ~ 69.2%),而原始评价的敏感性为94.5%(91.1 ~ 98.0%)。然而,Elicit识别了一些未被原始检索识别的纳入研究,平均准确率为41.8%(35.6-46.2%),高于原始评论的平均准确率7.55%(0.65-14.7%)。在本次评估时,Elicit的检索灵敏度不足以取代传统的文献检索。然而,在Elicit中搜索的高精度可能被证明对初步搜索有用,并且鉴定的独特研究意味着可以被研究人员用作有用的辅助语。结论虽然目前的Elicit搜索不够灵敏,不足以取代传统的搜索,但Elicit仍在不断改进,随着新的发展,应进行进一步的评估。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信