How does the credibility of vaccine information compare across traditional search engines and AI-based conversational agents?

IF 3.2 3区医学 Q1 PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH

Public Health Pub Date : 2025-08-01 DOI:10.1016/j.puhe.2025.105876

Amalie Dyda , Kristian Stanceski , Alexa Dakiniewich , Angela Pan , Adam G. Dunn

{"title":"How does the credibility of vaccine information compare across traditional search engines and AI-based conversational agents?","authors":"Amalie Dyda , Kristian Stanceski , Alexa Dakiniewich , Angela Pan , Adam G. Dunn","doi":"10.1016/j.puhe.2025.105876","DOIUrl":null,"url":null,"abstract":"<div><h3>Objectives</h3><div>Generative AI interfaces like ChatGPT offer a new way to access health information, but it is unclear if information presented is credible compared to traditional search engines. This study aimed to compare the credibility of vaccination information across generative AI interfaces and traditional search engines.</div></div><div><h3>Study design</h3><div>Cross sectional content analysis and comparison.</div></div><div><h3>Methods</h3><div>Questions were drawn from existing literature on common questions about vaccines and vaccination. Responses were retrieved in December 2023 by querying Google, Bing, Bard, ChatGPT 3.5, ChatGPT 4.0, and Claude AI. Credibility was measured using DISCERN and grade reading score was measured using standard measures via the SHeLL Editor.</div></div><div><h3>Results</h3><div>Across 12 questions, traditional search engines scored higher than generative AI in specific aspects of DISCERN, namely clarity of information sources (P < 0.0001), clarity of information recency (P < 0.0001) and provision of additional sources (P < 0.001). Generative AI interfaces performed better in relevance of information (P < 0.0001) and overall quality (P < 0.05).</div></div><div><h3>Conclusion</h3><div>Overall credibility of generative AI interfaces and traditional search engines is similar, but generative AI interfaces rarely provide sources and external links to high-quality information. In their current forms, generative AI interfaces may make information easy to read and appear credible, without providing typical credibility cues.</div></div>","PeriodicalId":49651,"journal":{"name":"Public Health","volume":"247 ","pages":"Article 105876"},"PeriodicalIF":3.2000,"publicationDate":"2025-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Public Health","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0033350625003221","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH","Score":null,"Total":0}

引用次数: 0

Abstract

Objectives

Generative AI interfaces like ChatGPT offer a new way to access health information, but it is unclear if information presented is credible compared to traditional search engines. This study aimed to compare the credibility of vaccination information across generative AI interfaces and traditional search engines.

Study design

Cross sectional content analysis and comparison.

Methods

Questions were drawn from existing literature on common questions about vaccines and vaccination. Responses were retrieved in December 2023 by querying Google, Bing, Bard, ChatGPT 3.5, ChatGPT 4.0, and Claude AI. Credibility was measured using DISCERN and grade reading score was measured using standard measures via the SHeLL Editor.

Results

Across 12 questions, traditional search engines scored higher than generative AI in specific aspects of DISCERN, namely clarity of information sources (P < 0.0001), clarity of information recency (P < 0.0001) and provision of additional sources (P < 0.001). Generative AI interfaces performed better in relevance of information (P < 0.0001) and overall quality (P < 0.05).

Conclusion

Overall credibility of generative AI interfaces and traditional search engines is similar, but generative AI interfaces rarely provide sources and external links to high-quality information. In their current forms, generative AI interfaces may make information easy to read and appear credible, without providing typical credibility cues.

查看原文本刊更多论文

疫苗信息的可信度如何在传统搜索引擎和基于人工智能的对话代理之间进行比较？

像ChatGPT这样的生成式人工智能界面提供了一种访问健康信息的新方式，但与传统搜索引擎相比，所呈现的信息是否可信尚不清楚。本研究旨在比较生成式人工智能界面和传统搜索引擎之间疫苗接种信息的可信度。研究设计横断面内容分析与比较。方法从现有文献中抽取有关疫苗和预防接种的常见问题。通过查询谷歌、Bing、Bard、ChatGPT 3.5、ChatGPT 4.0和Claude AI，在2023年12月检索了响应。使用DISCERN测量可信度，使用SHeLL Editor的标准测量方法测量年级阅读分数。在12个问题中，传统搜索引擎在DISCERN的特定方面得分高于生成式人工智能，即信息来源的清晰度(P <；0.0001)，信息近时性的清晰度(P <；0.0001)并提供额外来源(P <；0.001)。生成式AI界面在信息相关性方面表现更好(P <；0.0001)和整体质量(P <；0.05)。结论生成式AI界面的整体可信度与传统搜索引擎相似，但生成式AI界面很少提供高质量信息的来源和外部链接。在目前的形式下，生成式人工智能界面可能使信息易于阅读并显得可信，而不提供典型的可信度线索。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Public Health 医学-公共卫生、环境卫生与职业卫生

CiteScore

7.60

自引率

0.00%

发文量

280

审稿时长

37 days

期刊介绍： Public Health is an international, multidisciplinary peer-reviewed journal. It publishes original papers, reviews and short reports on all aspects of the science, philosophy, and practice of public health.