JMIR Medical Education最新文献

筛选
英文 中文
Citation Accuracy Challenges Posed by Large Language Models.
IF 3.2
JMIR Medical Education Pub Date : 2025-04-02 DOI: 10.2196/72998
Manlin Zhang, Tianyu Zhao
{"title":"Citation Accuracy Challenges Posed by Large Language Models.","authors":"Manlin Zhang, Tianyu Zhao","doi":"10.2196/72998","DOIUrl":"https://doi.org/10.2196/72998","url":null,"abstract":"","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"11 ","pages":"e72998"},"PeriodicalIF":3.2,"publicationDate":"2025-04-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143774438","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Extended Reality-Enhanced Mental Health Consultation Training: Quantitative Evaluation Study.
IF 3.2
JMIR Medical Education Pub Date : 2025-04-02 DOI: 10.2196/64619
Katherine Hiley, Zanib Bi-Mohammad, Luke Taylor, Rebecca Burgess-Dawson, Dominic Patterson, Devon Puttick-Whiteman, Christopher Gay, Janette Hiscoe, Chris Munsch, Sally Richardson, Mark Knowles-Lee, Celia Beecham, Neil Ralph, Arunangsu Chatterjee, Ryan Mathew, Faisal Mushtaq
{"title":"Extended Reality-Enhanced Mental Health Consultation Training: Quantitative Evaluation Study.","authors":"Katherine Hiley, Zanib Bi-Mohammad, Luke Taylor, Rebecca Burgess-Dawson, Dominic Patterson, Devon Puttick-Whiteman, Christopher Gay, Janette Hiscoe, Chris Munsch, Sally Richardson, Mark Knowles-Lee, Celia Beecham, Neil Ralph, Arunangsu Chatterjee, Ryan Mathew, Faisal Mushtaq","doi":"10.2196/64619","DOIUrl":"10.2196/64619","url":null,"abstract":"<p><strong>Background: </strong>The use of extended reality (XR) technologies in health care can potentially address some of the significant resource and time constraints related to delivering training for health care professionals. While substantial progress in realizing this potential has been made across several domains, including surgery, anatomy, and rehabilitation, the implementation of XR in mental health training, where nuanced humanistic interactions are central, has lagged.</p><p><strong>Objective: </strong>Given the growing societal and health care service need for trained mental health and care workers, coupled with the heterogeneity of exposure during training and the shortage of placement opportunities, we explored the feasibility and utility of a novel XR tool for mental health consultation training. Specifically, we set out to evaluate a training simulation created through collaboration among software developers, clinicians, and learning technologists, in which users interact with a virtual patient, \"Stacey,\" through a virtual reality or augmented reality head-mounted display. The tool was designed to provide trainee health care professionals with an immersive experience of a consultation with a patient presenting with perinatal mental health symptoms. Users verbally interacted with the patient, and a human instructor selected responses from a repository of prerecorded voice-acted clips.</p><p><strong>Methods: </strong>In a pilot experiment, we confirmed the face validity and usability of this platform for perinatal and primary care training with subject-matter experts. In our follow-up experiment, we delivered personalized 1-hour training sessions to 123 participants, comprising mental health nursing trainees, general practitioner doctors in training, and students in psychology and medicine. This phase involved a comprehensive evaluation focusing on usability, validity, and both cognitive and affective learning outcomes.</p><p><strong>Results: </strong>We found significant enhancements in learning metrics across all participant groups. Notably, there was a marked increase in understanding (P<.001) and motivation (P<.001), coupled with decreased anxiety related to mental health consultations (P<.001). There were also significant improvements to considerations toward careers in perinatal mental health (P<.001).</p><p><strong>Conclusions: </strong>Our findings show, for the first time, that XR can be used to provide an effective, standardized, and reproducible tool for trainees to develop their mental health consultation skills. We suggest that XR could provide a solution to overcoming the current resource challenges associated with equipping current and future health care professionals, which are likely to be exacerbated by workforce expansion plans.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"11 ","pages":"e64619"},"PeriodicalIF":3.2,"publicationDate":"2025-04-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143774442","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Authors' Reply: Citation Accuracy Challenges Posed by Large Language Models.
IF 3.2
JMIR Medical Education Pub Date : 2025-04-02 DOI: 10.2196/73698
Mohamad-Hani Temsah, Ayman Al-Eyadhy, Amr Jamal, Khalid Alhasan, Khalid H Malki
{"title":"Authors' Reply: Citation Accuracy Challenges Posed by Large Language Models.","authors":"Mohamad-Hani Temsah, Ayman Al-Eyadhy, Amr Jamal, Khalid Alhasan, Khalid H Malki","doi":"10.2196/73698","DOIUrl":"https://doi.org/10.2196/73698","url":null,"abstract":"","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"11 ","pages":"e73698"},"PeriodicalIF":3.2,"publicationDate":"2025-04-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143774435","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Media-Induced and Psychological Factors That Foster Empathy Through Virtual Reality in Nursing Education: 2×2 Between-Subjects Experimental Study.
IF 3.2
JMIR Medical Education Pub Date : 2025-03-31 DOI: 10.2196/59083
Kuo-Ting Huang, Zexin Ma, Lan Yao
{"title":"Media-Induced and Psychological Factors That Foster Empathy Through Virtual Reality in Nursing Education: 2×2 Between-Subjects Experimental Study.","authors":"Kuo-Ting Huang, Zexin Ma, Lan Yao","doi":"10.2196/59083","DOIUrl":"10.2196/59083","url":null,"abstract":"<p><strong>Background: </strong>Virtual reality (VR) has emerged as a promising tool in medical education, particularly for fostering critical skills such as empathy. However, how VR, combined with perspective-taking, influences affective empathy in nursing education remains underexplored.</p><p><strong>Objective: </strong>This study investigates the influence of VR and perspective-taking on affective empathy in nursing education, focusing on 4 psychological factors: perceived self-location, narrative transportation, emotional engagement, and affective empathy.</p><p><strong>Methods: </strong>A 2×2 between-subjects design was used, involving 69 nursing undergraduates from two Midwest universities. The participants engaged with a narrative-focused video game, That Dragon, Cancer, in either VR or non-VR conditions and from the perspective of either parents or clinicians.</p><p><strong>Results: </strong>VR significantly enhanced perceived self-location (P=.01), while adopting a clinician's perspective amplified emotional engagement (P=.03). However, VR did not significantly influence narrative transportation (P=.35). An interaction effect was found between the platform and player's perspective on narrative transportation (P=.04). Several indirect effects of media elements on affective empathy were observed via other psychological factors, though the direct effect of VR on affective empathy was not significant (P=.84).</p><p><strong>Conclusions: </strong>These findings underscore the potential of VR in medical education, suggesting that perspective-taking should be carefully considered when designing immersive learning experiences. The study advocates for broader integration of VR technologies into medical curricula to enhance instruction quality and patient-centered care.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"11 ","pages":"e59083"},"PeriodicalIF":3.2,"publicationDate":"2025-03-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11975256/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143754978","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Preclinical Medical Students' Perspectives and Experiences With Structured Web-Based English for Medical Purposes Courses: Cross-Sectional Study.
IF 3.2
JMIR Medical Education Pub Date : 2025-03-27 DOI: 10.2196/65779
Radhakrishnan Muthukumar, Isaraporn Thepwongsa, Poompong Sripa, Bangonsri Jindawong, Kamonwan Jenwitheesuk, Surapol Virasiri
{"title":"Preclinical Medical Students' Perspectives and Experiences With Structured Web-Based English for Medical Purposes Courses: Cross-Sectional Study.","authors":"Radhakrishnan Muthukumar, Isaraporn Thepwongsa, Poompong Sripa, Bangonsri Jindawong, Kamonwan Jenwitheesuk, Surapol Virasiri","doi":"10.2196/65779","DOIUrl":"10.2196/65779","url":null,"abstract":"<p><strong>Background: </strong>English for medical purposes (EMP) is essential for medical students as it serves as a foundational language for medical communication and education. However, students often undervalue its importance within the medical curriculum. Given their demanding schedules and workload, educational methods for EMP must align with their needs. Structured web-based learning offers flexibility and convenience, yet limited research has explored its exclusive application for EMP in undergraduate medical education.</p><p><strong>Objective: </strong>This study aimed to evaluate medical students' perspectives on structured web-based EMP courses and assess their impact on medical English proficiency using objective and subjective measures.</p><p><strong>Methods: </strong>Structured web-based EMP courses were developed based on evidence-based guidelines, addressing barriers to web-based learning during development and implementation. A cross-sectional study was conducted with 535 medical students who completed these courses. Data were collected via questionnaires, the learning management system, and the Khon Kaen University Medical English Test (KKUMET), which assessed proficiency in listening, reading, writing, and speaking. Data were analyzed using descriptive statistics.</p><p><strong>Results: </strong>Of the 535 students, 452 (84.5%) completed the survey. Participants reported confidence in reading (mean 4.11, SD 0.87), vocabulary (mean 4.04, SD 0.84), and listening skills (mean 4, SD 0.89), but lower confidence in writing skills (mean 3.46, SD 1.07). The KKUMET results showed statistically significant improvements in all 4 language skills after course completion (P<.001). The top-rated benefits of the courses were convenience (mean 4.77, SD 0.59), sufficient instruction (mean 4.5, SD 0.85), and clear content (mean 4.41, SD 0.80).</p><p><strong>Conclusions: </strong>Structured web-based EMP courses are relevant and well received by medical students. These courses significantly improve students' medical English proficiency, as evidenced by both subjective feedback and objective measures. Medical educators should consider integrating structured web-based EMP programs to better support students' language proficiency in medical contexts.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"11 ","pages":"e65779"},"PeriodicalIF":3.2,"publicationDate":"2025-03-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11967694/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143774453","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Knowledge Mapping and Global Trends in Simulation in Medical Education: Bibliometric and Visual Analysis.
IF 3.2
JMIR Medical Education Pub Date : 2025-03-26 DOI: 10.2196/71844
Hongjun Ba, Lili Zhang, Xiufang He, Shujuan Li
{"title":"Knowledge Mapping and Global Trends in Simulation in Medical Education: Bibliometric and Visual Analysis.","authors":"Hongjun Ba, Lili Zhang, Xiufang He, Shujuan Li","doi":"10.2196/71844","DOIUrl":"https://doi.org/10.2196/71844","url":null,"abstract":"<p><strong>Background: </strong>With the increasing recognition of the importance of simulation-based teaching in medical education, research in this field has developed rapidly. To comprehensively understand the research dynamics and trends in this area, we conducted an analysis of knowledge mapping and global trends.</p><p><strong>Objective: </strong>This study aims to reveal the research hotspots and development trends in the field of simulation-based teaching in medical education from 2004 to 2024 through bibliometric and visualization analyses.</p><p><strong>Methods: </strong>Using CiteSpace and VOSviewer, we conducted bibliometric and visualization analyses of 6743 articles related to simulation-based teaching in medical education, published in core journals from 2004 to 2024. The analysis included publication trends, contributions by countries and institutions, author contributions, keyword co-occurrence and clustering, and keyword bursts.</p><p><strong>Results: </strong>From 2004 to 2008, the number of articles published annually did not exceed 100. However, starting from 2009, the number increased year by year, reaching a peak of 850 articles in 2024, indicating rapid development in this research field. The United States, Canada, the United Kingdom, Australia, and China published the most articles. Harvard University emerged as a research hub with 1799 collaborative links, although the overall collaboration density was low. Among the 6743 core journal articles, a total of 858 authors were involved, with Lars Konge and Adam Dubrowski being the most prolific. However, collaboration density was low, and the collaboration network was relatively dispersed. A total of 812 common keywords were identified, forming 4189 links. The keywords \"medical education,\" \"education,\" and \"simulation\" had the highest frequency of occurrence. Cluster analysis indicated that \"cardiopulmonary resuscitation\" and \"surgical education\" were major research hotspots. From 2004 to 2024, a total of 20 burst keywords were identified, among which \"patient simulation,\" \"randomized controlled trial,\" \"clinical competence,\" and \"deliberate practice\" had high burst strength. In recent years, \"application of simulation in medical education,\" \"3D printing,\" \"augmented reality,\" and \"simulation training\" have become research frontiers.</p><p><strong>Conclusions: </strong>Research on the application of simulation-based teaching in medical education has become a hotspot, with expanding research areas and hotspots. Future research should strengthen interinstitutional collaboration and focus on the application of emerging technologies in simulation-based teaching.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"11 ","pages":"e71844"},"PeriodicalIF":3.2,"publicationDate":"2025-03-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143732119","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Ethical Use of Social Media and Sharing of Patient Information by Medical Students at a University Hospital in Saudi Arabia: Cross-Sectional Survey. 沙特阿拉伯一所大学医院医学生使用社交媒体和分享患者信息的道德问题:横断面调查
IF 3.2
JMIR Medical Education Pub Date : 2025-03-24 DOI: 10.2196/57812
Sara Farsi, Alaa Sabbahi, Deyala Sait, Raghad Kabli, Ghaliah Abduljabar
{"title":"Ethical Use of Social Media and Sharing of Patient Information by Medical Students at a University Hospital in Saudi Arabia: Cross-Sectional Survey.","authors":"Sara Farsi, Alaa Sabbahi, Deyala Sait, Raghad Kabli, Ghaliah Abduljabar","doi":"10.2196/57812","DOIUrl":"10.2196/57812","url":null,"abstract":"<p><strong>Background: </strong>Social media (SM) has become an integral part of many medical students' lives, blurring the lines between their personal and professional identities as many aspects of their medical careers appear online. Physicians must understand how to responsibly navigate these sites.</p><p><strong>Objective: </strong>This study aimed to identify how medical students use SM and their awareness and adherence to ethical guidelines of e-professionalism.</p><p><strong>Methods: </strong>This is a cross-sectional study delivered as an online voluntary survey to senior medical students at King AbdulAziz University Hospital in Jeddah, Saudi Arabia. We investigated how many students used SM, their privacy settings, their possible breaches of ethical standards, and their portrayal of their training institute online.</p><p><strong>Results: </strong>A total of 400/1546 (26%) senior medical students responded to our survey. Among the participants, 95/400 (24%) had public SM accounts, while 162/400 (41%) had both private and public accounts. As for breaches in e-professionalism, 11/400 (3%) participants posted a picture of a patient on SM without their permission, while 75/400 (20%) posted part of an excised organ or x-ray on SM without their permission, and 60/400 (16%) discussed a patient. With regards to sharing medical school information, 108/400 (29%) discussed an incident at their medical school, and 119/400 (31%) participants shared a lecture online without the presenter's permission. Approximately 66% of the participants reported that they were unaware if their institution had a professional code of conduct for SM use, and 259/371 (70%) did not receive training on the professional use of SM.</p><p><strong>Conclusions: </strong>Medical students must be taught to recognize inappropriate online behavior, understand their role as representatives of their medical school, and know the potential repercussions of unprofessional conduct on SM. This could be accomplished by providing workshops, regular seminars on e-professionalism, and including principles of SM conduct in existing ethics courses.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"11 ","pages":"e57812"},"PeriodicalIF":3.2,"publicationDate":"2025-03-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11957465/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143701710","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Creation of the ECHO Idaho Podcast: Tutorial and Pilot Assessment.
IF 3.2
JMIR Medical Education Pub Date : 2025-03-21 DOI: 10.2196/55313
Ryan Wiet, Madeline P Casanova, Jonathan D Moore, Sarah M Deming, Russell T Baker
{"title":"Creation of the ECHO Idaho Podcast: Tutorial and Pilot Assessment.","authors":"Ryan Wiet, Madeline P Casanova, Jonathan D Moore, Sarah M Deming, Russell T Baker","doi":"10.2196/55313","DOIUrl":"10.2196/55313","url":null,"abstract":"<p><strong>Background: </strong>Project ECHO (Extension for Community Health Outcomes) is an innovative program that uses videoconferencing technology to connect health care providers with experts. The model has been successful in reaching health care providers in rural and underserved areas and positively impacting clinical practice. ECHO Idaho, a replication partner, has developed programming that has increased knowledge and confidence of health care professionals throughout the state of Idaho, United States. Although the ECHO model has a demonstrated ability to recruit, educate, and train health care providers, barriers to attending Project ECHO continuing education (CE) programs remain. The asynchronous nature of podcasts could be used as an innovative medium to help address barriers to CE access that health care professionals face. The ECHO Idaho \"Something for the Pain\" podcast was developed to increase CE accessibility to rural and frontier providers, while upscaling their knowledge of and competence to treat and assess substance use disorders, pain, and behavioral health conditions.</p><p><strong>Objective: </strong>This paper describes the creation and preliminary assessment of the ECHO Idaho \"Something for the Pain\" podcast.</p><p><strong>Methods: </strong>Podcast episodes consisted of interviews with individuals as well as didactic lectures. Audio from these recordings were edited for content and length and then professionally reviewed by subject matter experts (eg, featured episode speakers). Target audiences consisted of health care providers and community members interested in behavioral health and substance use disorders. Metrics on podcast listeners were assessed using SoundCloud's RSS feed, continuing education survey completion, and iECHO.</p><p><strong>Results: </strong>The ECHO Idaho \"Something for the Pain\" podcast's inaugural season comprised 14 episodes with 626 minutes of CE material. The podcast series received a total of 2441 listens from individuals in 14 different cities across Idaho, and 63 health care providers listened and claimed CE credits. The largest professional group was social workers (n=22; 35%).</p><p><strong>Conclusions: </strong>We provide preliminary evidence that podcasts can be used to provide health care providers with opportunities to access CE material. Health care providers listened to and claimed CE credits from the ECHO Idaho \"Something for the Pain\" podcast. Project ECHO programs should consider creating podcasts as an additional platform for disseminating ECHO material.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"11 ","pages":"e55313"},"PeriodicalIF":3.2,"publicationDate":"2025-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11951813/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143674526","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Performance of Plug-In Augmented ChatGPT and Its Ability to Quantify Uncertainty: Simulation Study on the German Medical Board Examination.
IF 3.2
JMIR Medical Education Pub Date : 2025-03-21 DOI: 10.2196/58375
Julian Madrid, Philipp Diehl, Mischa Selig, Bernd Rolauffs, Felix Patricius Hans, Hans-Jörg Busch, Tobias Scheef, Leo Benning
{"title":"Performance of Plug-In Augmented ChatGPT and Its Ability to Quantify Uncertainty: Simulation Study on the German Medical Board Examination.","authors":"Julian Madrid, Philipp Diehl, Mischa Selig, Bernd Rolauffs, Felix Patricius Hans, Hans-Jörg Busch, Tobias Scheef, Leo Benning","doi":"10.2196/58375","DOIUrl":"10.2196/58375","url":null,"abstract":"<p><strong>Background: </strong>The GPT-4 is a large language model (LLM) trained and fine-tuned on an extensive dataset. After the public release of its predecessor in November 2022, the use of LLMs has seen a significant spike in interest, and a multitude of potential use cases have been proposed. In parallel, however, important limitations have been outlined. Particularly, current LLMs encounter limitations, especially in symbolic representation and accessing contemporary data. The recent version of GPT-4, alongside newly released plugin features, has been introduced to mitigate some of these limitations.</p><p><strong>Objective: </strong>Before this background, this work aims to investigate the performance of GPT-3.5, GPT-4, GPT-4 with plugins, and GPT-4 with plugins using pretranslated English text on the German medical board examination. Recognizing the critical importance of quantifying uncertainty for LLM applications in medicine, we furthermore assess this ability and develop a new metric termed \"confidence accuracy\" to evaluate it.</p><p><strong>Methods: </strong>We used GPT-3.5, GPT-4, GPT-4 with plugins, and GPT-4 with plugins and translation to answer questions from the German medical board examination. Additionally, we conducted an analysis to assess how the models justify their answers, the accuracy of their responses, and the error structure of their answers. Bootstrapping and CIs were used to evaluate the statistical significance of our findings.</p><p><strong>Results: </strong>This study demonstrated that available GPT models, as LLM examples, exceeded the minimum competency threshold established by the German medical board for medical students to obtain board certification to practice medicine. Moreover, the models could assess the uncertainty in their responses, albeit exhibiting overconfidence. Additionally, this work unraveled certain justification and reasoning structures that emerge when GPT generates answers.</p><p><strong>Conclusions: </strong>The high performance of GPTs in answering medical questions positions it well for applications in academia and, potentially, clinical practice. Its capability to quantify uncertainty in answers suggests it could be a valuable artificial intelligence agent within the clinical decision-making loop. Nevertheless, significant challenges must be addressed before artificial intelligence agents can be robustly and safely implemented in the medical domain.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"11 ","pages":"e58375"},"PeriodicalIF":3.2,"publicationDate":"2025-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11951815/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143674542","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Performance of ChatGPT-4 on Taiwanese Traditional Chinese Medicine Licensing Examinations: Cross-Sectional Study.
IF 3.2
JMIR Medical Education Pub Date : 2025-03-19 DOI: 10.2196/58897
Liang-Wei Tseng, Yi-Chin Lu, Liang-Chi Tseng, Yu-Chun Chen, Hsing-Yu Chen
{"title":"Performance of ChatGPT-4 on Taiwanese Traditional Chinese Medicine Licensing Examinations: Cross-Sectional Study.","authors":"Liang-Wei Tseng, Yi-Chin Lu, Liang-Chi Tseng, Yu-Chun Chen, Hsing-Yu Chen","doi":"10.2196/58897","DOIUrl":"10.2196/58897","url":null,"abstract":"<p><strong>Background: </strong>The integration of artificial intelligence (AI), notably ChatGPT, into medical education, has shown promising results in various medical fields. Nevertheless, its efficacy in traditional Chinese medicine (TCM) examinations remains understudied.</p><p><strong>Objective: </strong>This study aims to (1) assess the performance of ChatGPT on the TCM licensing examination in Taiwan and (2) evaluate the model's explainability in answering TCM-related questions to determine its suitability as a TCM learning tool.</p><p><strong>Methods: </strong>We used the GPT-4 model to respond to 480 questions from the 2022 TCM licensing examination. This study compared the performance of the model against that of licensed TCM doctors using 2 approaches, namely direct answer selection and provision of explanations before answer selection. The accuracy and consistency of AI-generated responses were analyzed. Moreover, a breakdown of question characteristics was performed based on the cognitive level, depth of knowledge, types of questions, vignette style, and polarity of questions.</p><p><strong>Results: </strong>ChatGPT achieved an overall accuracy of 43.9%, which was lower than that of 2 human participants (70% and 78.4%). The analysis did not reveal a significant correlation between the accuracy of the model and the characteristics of the questions. An in-depth examination indicated that errors predominantly resulted from a misunderstanding of TCM concepts (55.3%), emphasizing the limitations of the model with regard to its TCM knowledge base and reasoning capability.</p><p><strong>Conclusions: </strong>Although ChatGPT shows promise as an educational tool, its current performance on TCM licensing examinations is lacking. This highlights the need for enhancing AI models with specialized TCM training and suggests a cautious approach to utilizing AI for TCM education. Future research should focus on model improvement and the development of tailored educational applications to support TCM learning.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"11 ","pages":"e58897"},"PeriodicalIF":3.2,"publicationDate":"2025-03-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11939018/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143664655","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信