Evaluating reliability, quality, and readability of ChatGPT's nutritional recommendations for women with polycystic ovary syndrome.

Abstract

Patients with polycystic ovary syndrome (PCOS) often have many questions about nutrition and turn to chatbots such as Chat Generative Pretrained Transformer (ChatGPT) for advice. This study aims to evaluate the reliability, quality, and readability of ChatGPT's responses to nutrition-related questions asked by women with PCOS. Frequently asked nutrition-related questions from women with PCOS were reviewed in both Turkish and English. The reliability and quality of the answers were independently evaluated by 2 authors and a panel of 10 expert dietitians, using modified DISCERN and global quality score. Additionally, the readability of the answers was calculated using frequently used readability formulas. The mean modified DISCERN scores for English and Turkish versions were 27.6±0.87 and 27.2±0.87, respectively, indicating a fair level of reliability in the responses (16-31 points or 40%-79%). According to the global quality score, 100% of the responses in English and 90.9% of the responses in Turkish were rated as high quality. The readability of responses was classified as "difficult to read" with the readership levels assessed at college level and above for both English and Turkish. The correlation and regression analyses indicated no relationship between reliability, quality, and readability in English. However, a significant relationship was observed between quality and readability indexes in Turkish (P < .05). Our results suggest that ChatGPT's responses to nutrition-related questions about PCOS are generally of high quality, but improvements in both reliability and readability are still necessary. Although ChatGPT can offer general information and guidance on nutrition for PCOS, it should not be considered a substitute for personalized medical advice from health care professionals for effective management of the syndrome.