Evaluating Google Gemini’s AI generated responses to questions after breast reconstruction

Abstract

Background

The purpose of this study was to evaluate Google Gemini’s responses to common post-operative questions pertaining to breast reconstruction surgery.

Methods

Google Gemini AI was prompted with 14 common post-operative questions related to breast reconstruction surgery. Four experienced breast reconstructive surgeons and four advanced practice providers (APPs) evaluated the responses for accuracy, completeness, relevance, and overall quality on a 4-point Likert scale. Median scores were calculated and utilized as the final score. Responses were further categorized as accurate vs inaccurate, complete vs incomplete, relevant vs irrelevant, and high vs low quality. Readability was evaluated using the Fleisch-Kincaid reading scale.

Results

Attending surgeons classified 12/14 responses (86%) as accurate, 12/14 (86%) as complete, 13/14 (93%) as relevant, and 12/14 (86%) as high quality. APPs rated 11/14 responses (79%) as accurate, 12/14 (86%) as complete, 14 /14 (100%) as relevant, and 10/14 (71%) as high quality. APPs assigned lower median scores for overall quality than physicians (p=0.003). The mean Flesch-Kincaid readability score was 52.3.

Conclusion

Google Gemini provided relevant and complete responses to common post-operative questions pertaining to breast reconstruction. The “fairly difficult” readability score may pose challenges for certain patient populations. Differences in scores between physicians and APPs for “overall quality” may be due to higher levels of experience among physicians, allowing them to evaluate answers in a broader context. While Google Gemini demonstrates potential as a tool for patient education, patients should be advised to consult with clinicians for the most reliable and personalized medical advice.