Evaluating the accuracy and adequacy of ChatGPT in responding to queries of diabetes patients in primary healthcare

Abstract

Objective

This study evaluates the accuracy and adequacy of Chat Generative Pre-trained Transformer (ChatGPT) in responding to common queries formulated by primary care physicians based on their interactions with diabetic patients in primary healthcare settings.

Methods

Thirty-two frequently asked questions were identified by experienced primary care physicians and presented systematically to ChatGPT. Responses underwent evaluation by two endocrinology and metabolism physicians which utilized a 3-point Likert scale for accuracy (1, inaccurate; 2, partially accurate; 3, accurate) and a 6-point Likert scale for adequacy (1, completely inadequate to 6, completely adequate). Questions were categorized into groups including general information, diagnostic processes, treatment procedures, and complications.

Results

The median accuracy score was 3.0 (IQR, 3.0–3.0), and the adequacy score was 4.5 (IQR, 4.0–5.8). None of the questions received an inaccurate rating, and the lowest accuracy score assigned by both evaluators was 3. Significant agreement was observed between the evaluators, demonstrated by a weighted κ of 0.61 (p < .0001) for accuracy and substantial agreement with a weighted κ of 0.62 (p < 0.0001) for adequacy. The Kruskal–Wallis tests revealed no statistically significant differences among the groups for both accuracy (p = .71) and adequacy (p = .57).

Conclusions

ChatGPT demonstrated commendable accuracy and adequacy in addressing diabetes-related queries in primary healthcare.