How does the credibility of vaccine information compare across traditional search engines and AI-based conversational agents?

Objectives

Generative AI interfaces like ChatGPT offer a new way to access health information, but it is unclear if information presented is credible compared to traditional search engines. This study aimed to compare the credibility of vaccination information across generative AI interfaces and traditional search engines.

Study design

Cross sectional content analysis and comparison.

Methods

Questions were drawn from existing literature on common questions about vaccines and vaccination. Responses were retrieved in December 2023 by querying Google, Bing, Bard, ChatGPT 3.5, ChatGPT 4.0, and Claude AI. Credibility was measured using DISCERN and grade reading score was measured using standard measures via the SHeLL Editor.

Results

Across 12 questions, traditional search engines scored higher than generative AI in specific aspects of DISCERN, namely clarity of information sources (P < 0.0001), clarity of information recency (P < 0.0001) and provision of additional sources (P < 0.001). Generative AI interfaces performed better in relevance of information (P < 0.0001) and overall quality (P < 0.05).

Conclusion

Overall credibility of generative AI interfaces and traditional search engines is similar, but generative AI interfaces rarely provide sources and external links to high-quality information. In their current forms, generative AI interfaces may make information easy to read and appear credible, without providing typical credibility cues.