Leveraging artificial intelligence chatbots for anemia prevention: A comparative study of ChatGPT-3.5, copilot, and Gemini outputs against Google Search results

Abstract

Aim

This study evaluated the understandability, actionability, and readability of text on anemia generated by artificial intelligence (AI) chatbots.

Methods

This cross-sectional study compared texts generated by ChatGPT-3.5, Microsoft Copilot, and Google Gemini at three levels: “normal,” “6th grade,” and “PEMAT-P version.” Additionally, texts retrieved from the top eight Google Search results for relevant keywords were included for comparison. All texts were written in Japanese. The Japanese version of the PEMAT-P was used to assess understandability and actionability, while jReadability was used for readability. A systematic comparison was conducted to identify the strengths and weaknesses of each source.

Results

Texts generated by Gemini at the 6th-grade level (n = 26, 86.7 %) and PEMAT-P version (n = 27, 90.0 %), as well as ChatGPT-3.5 at the normal level (n = 21, 80.8 %), achieved significantly higher scores (≥70 %) for understandability and actionability compared to Google Search results (n = 17, 25.4 %, p < 0.001). For readability, Copilot and Gemini texts demonstrated significantly higher percentages of “very readable” to “somewhat difficult” levels than texts retrieved from Google Search (p = 0.000–0.007).

Innovation

This study is the first to objectively and quantitatively evaluate the understandability and actionability of educational materials on anemia prevention. By utilizing PEMAT-P and jReadability, the study demonstrated the superiority of Gemini in terms of understandability and readability through measurable data. This innovative approach highlights the potential of AI chatbots as a novel method for providing public health information and addressing health disparities.

Conclusion

AI-generated texts on anemia were found to be more readable and easier to understand than traditional web-based texts, with Gemini demonstrating the highest level of understandability. Moving forward, improvements in prompts will be necessary to enhance the integration of visual elements that encourage actionable responses in AI chatbots.