Feray Ekin Çiçek, Müşerref Ülker, Menekşe Özer, Yavuz Selim Kıyak
{"title":"ChatGPT versus expert feedback on clinical reasoning questions and their effect on learning: a randomized controlled trial.","authors":"Feray Ekin Çiçek, Müşerref Ülker, Menekşe Özer, Yavuz Selim Kıyak","doi":"10.1093/postmj/qgae170","DOIUrl":null,"url":null,"abstract":"<p><strong>Purpose: </strong>To evaluate the effectiveness of ChatGPT-generated feedback compared to expert-written feedback in improving clinical reasoning skills among first-year medical students.</p><p><strong>Methods: </strong>This is a randomized controlled trial conducted at a single medical school and involved 129 first-year medical students who were randomly assigned to two groups. Both groups completed three formative tests with feedback on urinary tract infections (UTIs; uncomplicated, complicated, pyelonephritis) over five consecutive days as a spaced repetition, receiving either expert-written feedback (control, n = 65) or ChatGPT-generated feedback (experiment, n = 64). Clinical reasoning skills were assessed using Key-Features Questions (KFQs) immediately after the intervention and 10 days later. Students' critical approach to artificial intelligence (AI) was also measured before and after disclosing the AI involvement in feedback generation.</p><p><strong>Results: </strong>There was no significant difference between the mean scores of the control (immediate: 78.5 ± 20.6 delayed: 78.0 ± 21.2) and experiment (immediate: 74.7 ± 15.1, delayed: 76.0 ± 14.5) groups in overall performance on Key-Features Questions (out of 120 points) immediately (P = .26) or after 10 days (P = .57), with small effect sizes. However, the control group outperformed the ChatGPT group in complicated urinary tract infection cases (P < .001). The experiment group showed a significantly higher critical approach to AI after disclosing, with medium-large effect sizes.</p><p><strong>Conclusions: </strong>ChatGPT-generated feedback can be an effective alternative to expert feedback in improving clinical reasoning skills in medical students, particularly in resource-constrained settings with limited expert availability. However, AI-generated feedback may lack the nuance needed for more complex cases, emphasizing the need for expert review. Additionally, exposure to the drawbacks in AI-generated feedback can enhance students' critical approach towards AI-generated educational content.</p>","PeriodicalId":20374,"journal":{"name":"Postgraduate Medical Journal","volume":" ","pages":""},"PeriodicalIF":3.6000,"publicationDate":"2024-12-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Postgraduate Medical Journal","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1093/postmj/qgae170","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"MEDICINE, GENERAL & INTERNAL","Score":null,"Total":0}
引用次数: 0
Abstract
Purpose: To evaluate the effectiveness of ChatGPT-generated feedback compared to expert-written feedback in improving clinical reasoning skills among first-year medical students.
Methods: This is a randomized controlled trial conducted at a single medical school and involved 129 first-year medical students who were randomly assigned to two groups. Both groups completed three formative tests with feedback on urinary tract infections (UTIs; uncomplicated, complicated, pyelonephritis) over five consecutive days as a spaced repetition, receiving either expert-written feedback (control, n = 65) or ChatGPT-generated feedback (experiment, n = 64). Clinical reasoning skills were assessed using Key-Features Questions (KFQs) immediately after the intervention and 10 days later. Students' critical approach to artificial intelligence (AI) was also measured before and after disclosing the AI involvement in feedback generation.
Results: There was no significant difference between the mean scores of the control (immediate: 78.5 ± 20.6 delayed: 78.0 ± 21.2) and experiment (immediate: 74.7 ± 15.1, delayed: 76.0 ± 14.5) groups in overall performance on Key-Features Questions (out of 120 points) immediately (P = .26) or after 10 days (P = .57), with small effect sizes. However, the control group outperformed the ChatGPT group in complicated urinary tract infection cases (P < .001). The experiment group showed a significantly higher critical approach to AI after disclosing, with medium-large effect sizes.
Conclusions: ChatGPT-generated feedback can be an effective alternative to expert feedback in improving clinical reasoning skills in medical students, particularly in resource-constrained settings with limited expert availability. However, AI-generated feedback may lack the nuance needed for more complex cases, emphasizing the need for expert review. Additionally, exposure to the drawbacks in AI-generated feedback can enhance students' critical approach towards AI-generated educational content.
期刊介绍：
Postgraduate Medical Journal is a peer reviewed journal published on behalf of the Fellowship of Postgraduate Medicine. The journal aims to support junior doctors and their teachers and contribute to the continuing professional development of all doctors by publishing papers on a wide range of topics relevant to the practicing clinician and teacher. Papers published in PMJ include those that focus on core competencies; that describe current practice and new developments in all branches of medicine; that describe relevance and impact of translational research on clinical practice; that provide background relevant to examinations; and papers on medical education and medical education research. PMJ supports CPD by providing the opportunity for doctors to publish many types of articles including original clinical research; reviews; quality improvement reports; editorials, and correspondence on clinical matters.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
