Comparison of Human and Artificial Intelligence (AI) in Writing and Rating Restorative Dentistry Essays.

Abstract

Aim: This study compared the writing quality of AI- and student-authored essays, evaluated faculty members' reliability to differentiate between essays authored by AI and students, and assessed the scoring accuracy between human and AI raters using a standardised rubric.

Methods: Four topics were selected from a preclinical operative and aesthetic dentistry course. Each topic was presented as four essays authored by two students and two AI tools (ChatGPT4 and Gemini) (N = 48). Then, the 16 essays were evaluated either by three blinded experts and two AI tool raters (ChatGPT4 and Gemini) using a modified Universal Science Writing Rubric. The Shapiro-Wilk W test assessed data normality. Kruskal-Wallis, Dunn's Pairwise, Wilcoxon Signed-rank, and Friedman tests analysed the writing performances and inter-rater reliabilities with a significance level of (α = 0.05).

Results: Significant differences were found in evaluating scientific content (Z = 9.28, p = 0.005) and interpreting scientific content (Z = 6.74, p = 0.021) between AI- and student-authored essays. ChatGPT4-authored essays differed significantly in scientific content from both Gemini- and student-authored essays, with further differences in interpretation between ChatGPT4- and student-authored essays (p = 0.011). Faculty members correctly identified 75% of essay authors. No significant differences were found between raters using ChatGPT4 or Gemini, while a marginally significant difference was observed between human raters in the overall score, though not in specific parameters.

Conclusions: Gemini's scores closely matched those of human-authored essays, aligning more with human raters than ChatGPT-4. AI's capacity emulated human writing, though differences are noticeable to trained faculty members. There are evident disparities in content quality and organisation between AI- and human-authored work.