JMIR Medical Education最新文献

{"title":"The Digital Determinants of Health: A Guide for Competency Development in Digital Care Delivery for Health Professions Trainees.","authors":"Katharine Lawrence, Defne L Levine","doi":"10.2196/54173","DOIUrl":"10.2196/54173","url":null,"abstract":"<p><strong>Unlabelled: </strong>Health care delivery is undergoing an accelerated period of digital transformation, spurred in part by the COVID-19 pandemic and the use of \"virtual-first\" care delivery models such as telemedicine. Medical education has responded to this shift with calls for improved digital health training, but there is as yet no universal understanding of the needed competencies, domains, and best practices for teaching these skills. In this paper, we argue that a \"digital determinants of health\" (DDoH) framework for understanding the intersections of health outcomes, technology, and training is critical to the development of comprehensive digital health competencies in medical education. Much like current social determinants of health models, the DDoH framework can be integrated into undergraduate, graduate, and professional education to guide training interventions as well as competency development and evaluation. We provide possible approaches to integrating this framework into training programs and explore priorities for future research in digitally-competent medical education.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":null,"pages":null},"PeriodicalIF":3.2,"publicationDate":"2024-08-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11376139/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142112987","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Exploring HTML5 Package Interactive Content in Supporting Learning Through Self-Paced Massive Open Online Courses on Healthy Aging: Mixed Methods Study.","authors":"Pratiwi Rahadiani, Aria Kekalih, Diantha Soemantri, Desak Gede Budi Krisnamurti","doi":"10.2196/45468","DOIUrl":"10.2196/45468","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;The rapidly aging population and the growth of geriatric medicine in the field of internal medicine are not supported by sufficient gerontological training in many health care disciplines. There is rising awareness about the education and training needed to adequately prepare health care professionals to address the needs of the older adult population. Massive open online courses (MOOCs) might be the best alternative method of learning delivery in this context. However, the diversity of MOOC participants poses a challenge for MOOC providers to innovate in developing learning content that suits the needs and characters of participants.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;The primary outcome of this study was to explore students' perceptions and acceptance of HTML5 package (H5P) interactive content in self-paced MOOCs and its association with students' characteristics and experience in using MOOCs.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;This study used a cross-sectional design, combining qualitative and quantitative approaches. Participants, predominantly general practitioners from various regions of Indonesia with diverse educational backgrounds and age groups, completed pretests, engaged with H5P interactive content, and participated in forum discussions and posttests. Data were retrieved from the online questionnaire attached to a selected MOOC course. Students' perceptions and acceptance of H5P interactive content were rated on a 6-point Likert scale from 1 (strongly disagree) to 6 (strongly agree). Data were analyzed using SPSS (IBM Corp) to examine demographics, computer literacy, acceptance, and perceptions of H5P interactive content. Quantitative analysis explored correlations, while qualitative analysis identified recurring themes from open-ended survey responses to determine students' perceptions.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;In total, 184 MOOC participants agreed to participate in the study. Students demonstrated positive perceptions and a high level of acceptance of integrating H5P interactive content within the self-paced MOOC. Analysis of mean (SD) value across all responses consistently revealed favorable scores (greater than 5), ranging from 5.18 (SD 0.861) to 5.45 (SD 0.659) and 5.28 (SD 0.728) to 5.52 (SD 0.627), respectively. This finding underscores widespread satisfaction and robust acceptance of H5P interactive content. Students found the H5P interactive content more satisfying and fun, easier to understand, more effective, and more helpful in improving learning outcomes than material in the form of common documents and learning videos. There is a significant correlation between computer literacy, students' acceptance, and students' perceptions.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Conclusions: &lt;/strong&gt;Students from various backgrounds showed a high level of acceptance and positive perceptions of leveraging H5P interactive content in the self-paced MOOC. The findings suggest potential new uses of H5P interactive content in","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":null,"pages":null},"PeriodicalIF":3.2,"publicationDate":"2024-08-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11377901/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141761457","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Integration of ChatGPT Into a Course for Medical Students: Explorative Study on Teaching Scenarios, Students' Perception, and Applications.","authors":"Anita V Thomae, Claudia M Witt, Jürgen Barth","doi":"10.2196/50545","DOIUrl":"10.2196/50545","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;Text-generating artificial intelligence (AI) such as ChatGPT offers many opportunities and challenges in medical education. Acquiring practical skills necessary for using AI in a clinical context is crucial, especially for medical education.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;This explorative study aimed to investigate the feasibility of integrating ChatGPT into teaching units and to evaluate the course and the importance of AI-related competencies for medical students. Since a possible application of ChatGPT in the medical field could be the generation of information for patients, we further investigated how such information is perceived by students in terms of persuasiveness and quality.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;ChatGPT was integrated into 3 different teaching units of a blended learning course for medical students. Using a mixed methods approach, quantitative and qualitative data were collected. As baseline data, we assessed students' characteristics, including their openness to digital innovation. The students evaluated the integration of ChatGPT into the course and shared their thoughts regarding the future of text-generating AI in medical education. The course was evaluated based on the Kirkpatrick Model, with satisfaction, learning progress, and applicable knowledge considered as key assessment levels. In ChatGPT-integrating teaching units, students evaluated videos featuring information for patients regarding their persuasiveness on treatment expectations in a self-experience experiment and critically reviewed information for patients written using ChatGPT 3.5 based on different prompts.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;A total of 52 medical students participated in the study. The comprehensive evaluation of the course revealed elevated levels of satisfaction, learning progress, and applicability specifically in relation to the ChatGPT-integrating teaching units. Furthermore, all evaluation levels demonstrated an association with each other. Higher openness to digital innovation was associated with higher satisfaction and, to a lesser extent, with higher applicability. AI-related competencies in other courses of the medical curriculum were perceived as highly important by medical students. Qualitative analysis highlighted potential use cases of ChatGPT in teaching and learning. In ChatGPT-integrating teaching units, students rated information for patients generated using a basic ChatGPT prompt as \"moderate\" in terms of comprehensibility, patient safety, and the correct application of communication rules taught during the course. The students' ratings were considerably improved using an extended prompt. The same text, however, showed the smallest increase in treatment expectations when compared with information provided by humans (patient, clinician, and expert) via videos.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Conclusions: &lt;/strong&gt;This study offers valuable insights into integrating the development of AI competencies into a ","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":null,"pages":null},"PeriodicalIF":3.2,"publicationDate":"2024-08-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11360267/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142037198","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Newly Qualified Canadian Nurses' Experiences With Digital Health in the Workplace: Comparative Qualitative Analysis.","authors":"Manal Kleib, Antonia Arnaert, Lynn M Nagle, Rebecca Sugars, Daniel da Costa","doi":"10.2196/53258","DOIUrl":"10.2196/53258","url":null,"abstract":"<p><strong>Background: </strong>Clinical practice settings have increasingly become dependent on the use of digital or eHealth technologies such as electronic health records. It is vitally important to support nurses in adapting to digitalized health care systems; however, little is known about nursing graduates' experiences as they transition to the workplace.</p><p><strong>Objective: </strong>This study aims to (1) describe newly qualified nurses' experiences with digital health in the workplace, and (2) identify strategies that could help support new graduates' transition and practice with digital health.</p><p><strong>Methods: </strong>An exploratory descriptive qualitative design was used. A total of 14 nurses from Eastern and Western Canada participated in semistructured interviews and data were analyzed using inductive content analysis.</p><p><strong>Results: </strong>Three themes were identified: (1) experiences before becoming a registered nurse, (2) experiences upon joining the workplace, and (3) suggestions for bridging the gap in transition to digital health practice. Findings revealed more similarities than differences between participants with respect to gaps in digital health education, technology-related challenges, and their influence on nursing practice.</p><p><strong>Conclusions: </strong>Digital health is the foundation of contemporary health care; therefore, comprehensive education during nursing school and throughout professional nursing practice, as well as organizational support and policy, are critical pillars. Health systems investing in digital health technologies must create supportive work environments for nurses to thrive in technologically rich environments and increase their capacity to deliver the digital health future.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":null,"pages":null},"PeriodicalIF":3.2,"publicationDate":"2024-08-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11369539/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142005452","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Language Model-Powered Simulated Patient With Automated Feedback for History Taking: Prospective Study.","authors":"Friederike Holderried, Christian Stegemann-Philipps, Anne Herrmann-Werner, Teresa Festl-Wietek, Martin Holderried, Carsten Eickhoff, Moritz Mahling","doi":"10.2196/59213","DOIUrl":"10.2196/59213","url":null,"abstract":"<p><strong>Background: </strong>Although history taking is fundamental for diagnosing medical conditions, teaching and providing feedback on the skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged as educational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing their realism and potential to provide feedback.</p><p><strong>Objective: </strong>In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students' performance in history taking with a simulated patient.</p><p><strong>Methods: </strong>We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients' responses and provide immediate feedback on the comprehensiveness of the students' history taking. Students' interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback.</p><p><strong>Results: </strong>Most of the study's participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4's role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed \"almost perfect\" agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model's assessments were overly specific or diverged from human judgement.</p><p><strong>Conclusions: </strong>The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":null,"pages":null},"PeriodicalIF":3.2,"publicationDate":"2024-08-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11364946/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141989103","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Bridging the Gap: A Mixed-Method Study on Gender and Sexuality Awareness in Medical Education and Practice.","authors":"Rola Khamisy-Farah, Eden Biras, Rabie Shehadeh, Ruba Tuma, Hisham Atwan, Anna Siri, Manlio Converti, Francesco Chirico, Lukasz Szarpak, Carlo Biz, Raymond Farah, Nicola Bragazzi","doi":"10.2196/59009","DOIUrl":"https://doi.org/10.2196/59009","url":null,"abstract":"<p><strong>Background: </strong>The integration of gender and sexuality awareness in healthcare is increasingly recognized as vital for patient outcomes. Despite this, there is a notable lack of comprehensive data on the current state of physicians' training and perceptions in these areas, leading to a gap in targeted educational interventions and optimal healthcare delivery.</p><p><strong>Objective: </strong>The study's aim was to explore the experiences and perceptions of attending and resident physicians regarding the inclusion of gender and sexuality content in medical school curricula and professional practice in Israel.</p><p><strong>Methods: </strong>This cross-sectional survey targeted a diverse group of physicians across various specializations and experience levels. Distributed through Israeli medical associations and professional networks, it included sections on experiences with gender and sexuality content, perceptions of knowledge, the impact of medical school curricula on professional capabilities, and views on integrating gender medicine in medical education. Descriptive and correlational analyses, along with gender-based and medical status-based comparisons, were employed, complemented and enhanced by qualitative analysis of participants' replies.</p><p><strong>Results: </strong>The survey, encompassing 189 respondents, revealed low-to-moderate exposure to gender and sexuality content in medical school curricula, with a similar perception of preparedness. A need for more comprehensive training was widely recognized. The majority valued training in these areas for enhancing professional capabilities, identifying ten essential gender-related knowledge areas. The preference for integrating gender medicine throughout medical education was significant. Gender-based analysis indicated variations in exposure and perceptions.</p><p><strong>Conclusions: </strong>The study highlights a crucial need for the inclusion of gender and sexuality awareness in medical education and practice. It suggests the necessity for curriculum development, targeted training programs, policy advocacy, mentorship initiatives, and research to evaluate the effectiveness of these interventions. The findings serve as a foundation for future directions in medical education, aiming for a more inclusive, aware, and prepared medical workforce.</p><p><strong>Clinicaltrial: </strong></p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":null,"pages":null},"PeriodicalIF":3.2,"publicationDate":"2024-08-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141996614","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Reforming China's Secondary Vocational Medical Education: Adapting to the Challenges and Opportunities of the AI Era.","authors":"Wenting Tong, Xiaowen Zhang, Haiping Zeng, Jianping Pan, Chao Gong, Hui Zhang","doi":"10.2196/48594","DOIUrl":"10.2196/48594","url":null,"abstract":"<p><strong>Unlabelled: </strong>China's secondary vocational medical education is essential for training primary health care personnel and enhancing public health responses. This education system currently faces challenges, primarily due to its emphasis on knowledge acquisition that overshadows the development and application of skills, especially in the context of emerging artificial intelligence (AI) technologies. This article delves into the impact of AI on medical practices and uses this analysis to suggest reforms for the vocational medical education system in China. AI is found to significantly enhance diagnostic capabilities, therapeutic decision-making, and patient management. However, it also brings about concerns such as potential job losses and necessitates the adaptation of medical professionals to new technologies. Proposed reforms include a greater focus on critical thinking, hands-on experiences, skill development, medical ethics, and integrating humanities and AI into the curriculum. These reforms require ongoing evaluation and sustained research to effectively prepare medical students for future challenges in the field.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":null,"pages":null},"PeriodicalIF":3.2,"publicationDate":"2024-08-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11337726/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141989105","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Teaching Digital Medicine to Undergraduate Medical Students with an Interprofessional and Interdisciplinary Approach: A Proof of Concept Study.","authors":"Annabelle Mielitz, Ulf Kulau, Lucas Bublitz, Anja Bittner, Hendrik Friederichs, Urs-Vito Albrecht","doi":"10.2196/56787","DOIUrl":"https://doi.org/10.2196/56787","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;An integration of digital medicine into medical education can help future doctors to shape the digital transformation of medicine.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;A newly developed course for teaching digital medicine (Bielefeld model) is described and evaluated for the first time.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;The course was held with undergraduate medical students at Medical School OWL at Bielefeld University, Germany, in 2023, and evaluated via pre-post surveys. The subjective and objective achievement of superordinate learning objectives and the objective achievement of subordinate learning objectives of the course, course design and course importance were evaluated utilizing five-point Likert scales (1=\"strongly disagree\", 5=\"strongly agree\"), a multiple choice format for reasons of absence, and open comments. The superordinate objectives comprise the understanding of factors driving the implementation of digital medical products and processes (1), the application of this knowledge to a project (2), and the empowerment to design such solutions (3) in the future. The subordinate objectives comprise competencies related to the first superordinate objective.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;10 undergraduate medical students (male: 4, female: 6, mean age 21.7 years, SD: 2.1 years) evaluated the course. The superordinate objectives were achieved well to very well: the medians for the objective achievement were 4 scale units (su) (IQR 4-5 su), 4 su (IQR 3-5 su) and 4 su (IQR 4-4 su) for the first, second and third objective and the medians for the subjective achievement of the first, second and third objective were 4 su (IQR 3-4 su), 4.5 su (IQR 3-5 su) and 4 su (IQR 3-5 su). Participants mastered the subordinate objectives, averagely, better after the course than before (pre-survey median:2.5 su (IQR 2-3 su), post-survey median: 4 su (IQR 3-4 su)). The course concept was rated as highly suitable for achieving the superordinate objectives (medians: 5 su (IQR 4-5 su) for the first, second, and third objective). On average, the students highly liked the course (median: 5 su (IQR 4-5 su)) and gained a benefit from (median: 4.5 su (IQR 4-5 su)). All students fully agreed that the teaching staff was a strength of the course. The category \"Positive feedback on the course or positive personal experience with the course\" received most comments.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Conclusions: &lt;/strong&gt;The course framework exhibits promise in attaining learning objectives within the realm of digital medicine, notwithstanding the constraint of limited interpretability arising from a small sample size and further limitations. It aligns with insights derived from teaching and learning research and the domain of digital medicine, albeit with identifiable areas for enhancement. A literature review indicates a dearth of publications pertaining to analogous courses in Germany. Future investigations should entail a more exhaustive evaluation of the co","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":null,"pages":null},"PeriodicalIF":3.2,"publicationDate":"2024-08-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142074128","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Impact of a New Gynecologic Oncology Hashtag During Virtual-Only ASCO Annual Meetings: An X (Twitter) Social Network Analysis.","authors":"Geetu Bhandoria, Esra Bilir, Christina Uwins, Josep Vidal-Alaball, Aïna Fuster-Casanovas, Wasim Ahmed","doi":"10.2196/45291","DOIUrl":"10.2196/45291","url":null,"abstract":"<p><strong>Background: </strong>Official conference hashtags are commonly used to promote tweeting and social media engagement. The reach and impact of introducing a new hashtag during an oncology conference have yet to be studied. The American Society of Clinical Oncology (ASCO) conducts an annual global meeting, which was entirely virtual due to the COVID-19 pandemic in 2020 and 2021.</p><p><strong>Objective: </strong>This study aimed to assess the reach and impact (in the form of vertices and edges generated) and X (formerly Twitter) activity of the new hashtags #goASCO20 and #goASCO21 in the ASCO 2020 and 2021 virtual conferences.</p><p><strong>Methods: </strong>New hashtags (#goASCO20 and #goASCO21) were created for the ASCO virtual conferences in 2020 and 2021 to help focus gynecologic oncology discussion at the ASCO meetings. Data were retrieved using these hashtags (#goASCO20 for 2020 and #goASCO21 for 2021). A social network analysis was performed using the NodeXL software application.</p><p><strong>Results: </strong>The hashtags #goASCO20 and #goASCO21 had similar impacts on the social network. Analysis of the reach and impact of the individual hashtags found #goASCO20 to have 150 vertices and 2519 total edges and #goASCO20 to have 174 vertices and 2062 total edges. Mentions and tweets between 2020 and 2021 were also similar. The circles representing different users were spatially arranged in a more balanced way in 2021. Tweets using the #goASCO21 hashtag received significantly more responses than tweets using #goASCO20 (75 times in 2020 vs 360 times in 2021; z value=16.63 and P<.001). This indicates increased engagement in the subsequent year.</p><p><strong>Conclusions: </strong>Introducing a gynecologic oncology specialty-specific hashtag (#goASCO20 and #goASCO21) that is related but different from the official conference hashtag (#ASCO20 and #ASCO21) helped facilitate discussion on topics of interest to gynecologic oncologists during a virtual pan-oncology meeting. This impact was visible in the social network analysis.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":null,"pages":null},"PeriodicalIF":3.2,"publicationDate":"2024-08-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11339558/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141989104","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Influence of Model Evolution and System Roles on ChatGPT's Performance in Chinese Medical Licensing Exams: Comparative Study.","authors":"Shuai Ming, Qingge Guo, Wenjun Cheng, Bo Lei","doi":"10.2196/52784","DOIUrl":"10.2196/52784","url":null,"abstract":"<p><strong>Background: </strong>With the increasing application of large language models like ChatGPT in various industries, its potential in the medical domain, especially in standardized examinations, has become a focal point of research.</p><p><strong>Objective: </strong>The aim of this study is to assess the clinical performance of ChatGPT, focusing on its accuracy and reliability in the Chinese National Medical Licensing Examination (CNMLE).</p><p><strong>Methods: </strong>The CNMLE 2022 question set, consisting of 500 single-answer multiple choices questions, were reclassified into 15 medical subspecialties. Each question was tested 8 to 12 times in Chinese on the OpenAI platform from April 24 to May 15, 2023. Three key factors were considered: the version of GPT-3.5 and 4.0, the prompt's designation of system roles tailored to medical subspecialties, and repetition for coherence. A passing accuracy threshold was established as 60%. The χ2 tests and κ values were employed to evaluate the model's accuracy and consistency.</p><p><strong>Results: </strong>GPT-4.0 achieved a passing accuracy of 72.7%, which was significantly higher than that of GPT-3.5 (54%; P<.001). The variability rate of repeated responses from GPT-4.0 was lower than that of GPT-3.5 (9% vs 19.5%; P<.001). However, both models showed relatively good response coherence, with κ values of 0.778 and 0.610, respectively. System roles numerically increased accuracy for both GPT-4.0 (0.3%-3.7%) and GPT-3.5 (1.3%-4.5%), and reduced variability by 1.7% and 1.8%, respectively (P>.05). In subgroup analysis, ChatGPT achieved comparable accuracy among different question types (P>.05). GPT-4.0 surpassed the accuracy threshold in 14 of 15 subspecialties, while GPT-3.5 did so in 7 of 15 on the first response.</p><p><strong>Conclusions: </strong>GPT-4.0 passed the CNMLE and outperformed GPT-3.5 in key areas such as accuracy, consistency, and medical subspecialty expertise. Adding a system role insignificantly enhanced the model's reliability and answer coherence. GPT-4.0 showed promising potential in medical education and clinical practice, meriting further study.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":null,"pages":null},"PeriodicalIF":3.2,"publicationDate":"2024-08-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11336778/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141976840","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}