Can artificial intelligence read between the lines: Utilizing ChatGPT to evaluate medical students' implicit attitudes towards doctor-patient relationship.

IF 3.3 2区教育学 Q1 EDUCATION, SCIENTIFIC DISCIPLINES

Medical Teacher Pub Date : 2025-06-11 DOI:10.1080/0142159X.2025.2515971

Wenqi Geng, Yinan Jiang, Wei Zhai, Xiaohui Zhao, Qing Zhao, Jianqiang Li, Jinya Cao, Lili Shi

{"title":"Can artificial intelligence read between the lines: Utilizing ChatGPT to evaluate medical students' implicit attitudes towards doctor-patient relationship.","authors":"Wenqi Geng, Yinan Jiang, Wei Zhai, Xiaohui Zhao, Qing Zhao, Jianqiang Li, Jinya Cao, Lili Shi","doi":"10.1080/0142159X.2025.2515971","DOIUrl":null,"url":null,"abstract":"Purpose: To explore ChatGPT's utility in evaluating medical students' implicit attitudes toward the doctor-patient relationship (DPR).Materials and methods: This study analyzed interview transcripts from 10 medical students, categorizing implicit DPR attitudes into Care and Share dimensions, each with 5 levels. We first assessed ChatGPT's ability to identify DPR-related textual content, then compared grading results from experts, ChatGPT, and participants' self-evaluations. Finally, experts evaluated ChatGPT's performance acceptability.Results: ChatGPT annotated fewer DPR-related segments than human experts. In grading, pre-course scores from experts and ChatGPT were comparable but lower than self-assessments. Post-course, expert scores were lower than ChatGPT's and further below self-assessments. ChatGPT achieved an accuracy of 0.84-0.85, precision of 0.81-0.85, recall of 0.84-0.85, and F1 score of 0.82-0.84 for attitude classification, with an average acceptability score of 3.9/5.Conclusions: Large language models (LLMs) demonstrated high consistency with human experts in judging implicit attitudes. Future research should optimize LLMs and replicate this framework across diverse contexts with larger samples.","PeriodicalId":18643,"journal":{"name":"Medical Teacher","volume":" ","pages":"1-8"},"PeriodicalIF":3.3000,"publicationDate":"2025-06-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Medical Teacher","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1080/0142159X.2025.2515971","RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

Purpose: To explore ChatGPT's utility in evaluating medical students' implicit attitudes toward the doctor-patient relationship (DPR).

Materials and methods: This study analyzed interview transcripts from 10 medical students, categorizing implicit DPR attitudes into Care and Share dimensions, each with 5 levels. We first assessed ChatGPT's ability to identify DPR-related textual content, then compared grading results from experts, ChatGPT, and participants' self-evaluations. Finally, experts evaluated ChatGPT's performance acceptability.

Results: ChatGPT annotated fewer DPR-related segments than human experts. In grading, pre-course scores from experts and ChatGPT were comparable but lower than self-assessments. Post-course, expert scores were lower than ChatGPT's and further below self-assessments. ChatGPT achieved an accuracy of 0.84-0.85, precision of 0.81-0.85, recall of 0.84-0.85, and F1 score of 0.82-0.84 for attitude classification, with an average acceptability score of 3.9/5.

Conclusions: Large language models (LLMs) demonstrated high consistency with human experts in judging implicit attitudes. Future research should optimize LLMs and replicate this framework across diverse contexts with larger samples.

查看原文本刊更多论文

人工智能能读懂言外之意吗？利用ChatGPT评估医学生对医患关系的内隐态度。

目的：探讨ChatGPT在评价医学生医患关系内隐态度中的应用价值。材料与方法：本研究对10名医学生的访谈记录进行分析，将内隐DPR态度分为“关心”和“分享”两个维度，每个维度有5个层次。我们首先评估了ChatGPT识别pr相关文本内容的能力，然后比较了专家、ChatGPT和参与者自我评估的评分结果。最后，专家评估了ChatGPT的性能可接受性。结果：ChatGPT比人类专家注释的pr相关片段更少。在评分方面，专家和ChatGPT的课前分数相当，但低于自我评估。课程结束后，专家得分低于ChatGPT，进一步低于自我评估。ChatGPT的姿态分类正确率为0.84 ~ 0.85，精密度为0.81 ~ 0.85，召回率为0.84 ~ 0.85，F1得分为0.82 ~ 0.84，平均可接受得分为3.9/5。结论：大型语言模型在判断内隐态度方面与人类专家具有较高的一致性。未来的研究应该优化llm，并在不同的背景下用更大的样本复制这个框架。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Medical Teacher 医学-卫生保健

CiteScore

7.80

自引率

8.50%

发文量

396

审稿时长

3-6 weeks

期刊介绍： Medical Teacher provides accounts of new teaching methods, guidance on structuring courses and assessing achievement, and serves as a forum for communication between medical teachers and those involved in general education. In particular, the journal recognizes the problems teachers have in keeping up-to-date with the developments in educational methods that lead to more effective teaching and learning at a time when the content of the curriculum—from medical procedures to policy changes in health care provision—is also changing. The journal features reports of innovation and research in medical education, case studies, survey articles, practical guidelines, reviews of current literature and book reviews. All articles are peer reviewed.