Journal of Educational Evaluation for Health Professions最新文献

{"title":"Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: a descriptive study","authors":"Sun Huh","doi":"10.3352/jeehp.2023.20.1","DOIUrl":"https://doi.org/10.3352/jeehp.2023.20.1","url":null,"abstract":"<p><p>This study aimed to compare the knowledge and interpretation ability of ChatGPT, a language model of artificial general intelligence, with those of medical students in Korea by administering a parasitology examination to both ChatGPT and medical students. The examination consisted of 79 items and was administered to ChatGPT on January 1, 2023. The examination results were analyzed in terms of ChatGPT’s overall performance score, its correct answer rate by the items’ knowledge level, and the acceptability of its explanations of the items. ChatGPT’s performance was lower than that of the medical students, and ChatGPT’s correct answer rate was not related to the items’ knowledge level. However, there was a relationship between acceptable explanations and correct answers. In conclusion, ChatGPT’s knowledge and interpretation ability for this parasitology examination were not yet comparable to those of medical students in Korea.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"20 ","pages":"1"},"PeriodicalIF":4.4,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9905868/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9301278","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Effect of an interprofessional simulation program on patient safety competencies of healthcare professionals in Switzerland: a before and after study.","authors":"Sylvain Boloré, Thomas Fassier, Nicolas Guirimand","doi":"10.3352/jeehp.2023.20.25","DOIUrl":"10.3352/jeehp.2023.20.25","url":null,"abstract":"<p><strong>Purpose: </strong>This study aimed to identify the effects of a 12-week interprofessional simulation program, operated between February 2020 and January 2021, on the patient safety competencies of healthcare professionals in Switzerland.</p><p><strong>Methods: </strong>The simulation training was based on 2 scenarios of hospitalized patients with septic shock and respiratory failure, and trainees were expected to demonstrate patient safety competencies. A single-group before and after study was conducted after the intervention-simulation program, using a measurement tool (the Health Professional Education in Patient Safety Survey) to measure the perceived competencies of physicians, nurses, and nursing assistants. Out of 57 participants, 37 answered the questionnaire surveys 4 times: 48 hours before the training, followed by post-surveys at 24 hours, 6 weeks, and 12 weeks after the training. The linear mixed effect model was applied for the analysis.</p><p><strong>Results: </strong>Four components out of 6 perceived patient safety competencies improved at 6 weeks but returned to a similar level before training at 12 weeks. Competencies of \"communicating effectively,\" \"managing safety risks,\" \"understanding human and environmental factors that influence patient safety,\" and \"recognize and respond to remove immediate risks of harm\" are statistically significant both overall and in the comparison between before the training and 6 weeks after the training.</p><p><strong>Conclusion: </strong>Interprofessional simulation programs contributed to developing some areas of patient safety competencies of healthcare professionals, but only for a limited time. Interprofessional simulation programs should be repeated and combined with other forms of support, including case discussions and debriefings, to ensure lasting effects.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"20 ","pages":"25"},"PeriodicalIF":9.3,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10519772/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10481364","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Assessment of the viability of integrating virtual reality programs in practical tests for the Korean Radiological Technologists Licensing Examination: a survey study","authors":"Hye Min Park, Eun Seong Kim, Deok Mun Kwon, Pyong Kon Cho, Seoung Hwan Kim, Ki Baek Lee, Seong Hu Kim, Moon Il Bong, Won Seok Yang, Jin Eui Kim, Gi Bong Kang, Yong Su Yoon, Jung Su Kim","doi":"10.3352/jeehp.2023.20.33","DOIUrl":"10.3352/jeehp.2023.20.33","url":null,"abstract":"<p><strong>Purpose: </strong>The objective of this study was to assess the feasibility of incorporating virtual reality/augmented reality (VR/AR) programs into practical tests administered as part of the Korean Radiological Technologists Licensing Examination (KRTLE). This evaluation is grounded in a comprehensive survey that targeted enrolled students in departments of radiology across the nation.</p><p><strong>Methods: </strong>In total, 682 students from radiology departments across the nation were participants in the survey. An online survey platform was used, and the questionnaire was structured into 5 distinct sections and 27 questions. A frequency analysis for each section of the survey was conducted using IBM SPSS ver. 27.0.</p><p><strong>Results: </strong>Direct or indirect exposure to VR/AR content was reported by 67.7% of all respondents. Furthermore, 55.4% of the respondents expressed that VR/AR could be integrated into their classes, which signified a widespread acknowledgment of VR among the students. With regards to the integration of a VR/AR or mixed reality program into the practical tests for purposes of the KRTLE, a substantial amount of the respondents (57.3%) exhibited a positive inclination and recommended its introduction.</p><p><strong>Conclusion: </strong>The application of VR/AR programs within practical tests of the KRTLE will be used as an alternative for evaluating clinical examination procedures and validating job skills.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"20 ","pages":"33"},"PeriodicalIF":4.4,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10762238/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138446568","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Development of a character qualities test for medical students in Korea using polytomous item response theory and factor analysis: a preliminary scale development study","authors":"Yera Hur,&nbsp;Dong Gi Seo","doi":"10.3352/jeehp.2023.20.20","DOIUrl":"https://doi.org/10.3352/jeehp.2023.20.20","url":null,"abstract":"<p><strong>Purpose: </strong>This study aimed to develop a test scale to measure the character qualities of medical students as a follow-up study on the 8 core character qualities revealed in a previous report.</p><p><strong>Methods: </strong>In total, 160 preliminary items were developed to measure 8 core character qualities. Twenty questions were assigned to each quality, and a questionnaire survey was conducted among 856 students in 5 medical schools in Korea. Using the partial credit model, polytomous item response theory analysis was carried out to analyze the goodness-of-fit, followed by exploratory factor analysis. Finally, confirmatory factor and reliability analyses were conducted with the final selected items.</p><p><strong>Results: </strong>The preliminary items for the 8 core character qualities were administered to the participants. Data from 767 students were included in the final analysis. Of the 160 preliminary items, 25 were removed by classical test theory analysis and 17 more by polytomous item response theory assessment. A total of 118 items and sub-factors were selected for exploratory factor analysis. Finally, 79 items were selected, and the validity and reliability were confirmed through confirmatory factor analysis and intra-item relevance analysis.</p><p><strong>Conclusion: </strong>The character qualities test scale developed through this study can be used to measure the character qualities corresponding to the educational goals and visions of individual medical schools in Korea. Furthermore, this measurement tool can serve as primary data for developing character qualities tools tailored to each medical school’s vision and educational goals.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"20 ","pages":"20"},"PeriodicalIF":4.4,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10356546/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9839665","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Effect of a smartphone-based online electronic logbook to evaluate the clinical skills of nurse anesthesia students in Iran: a randomized controlled study.","authors":"Ali Khalafi,&nbsp;Nahid Jamshidi,&nbsp;Nasrin Khajeali,&nbsp;Saeed Ghanbari","doi":"10.3352/jeehp.2023.20.10","DOIUrl":"https://doi.org/10.3352/jeehp.2023.20.10","url":null,"abstract":"<p><strong>Purpose: </strong>This study was conducted to evaluate a smartphone-based online electronic logbook used to assess the clinical skills of nurse anesthesia students in Iran.</p><p><strong>Methods: </strong>This randomized controlled study was conducted after tool development at Ahvaz Jundishapur University of Medical Sciences in Ahvaz, Iran from January 2022 to December 2022. The online electronic logbook involved in this study was an Android-compatible application used to evaluate the clinical skills of nurse anesthesia students. In the implementation phase, the online electronic logbook was piloted for 3 months in anesthesia training in comparison with a paper logbook. For this purpose, 49 second- and third-year anesthesia nursing students selected using the census method were assigned to intervention (online electronic logbook) and control (paper logbook) groups. The online electronic logbook and paper logbook were compared in terms of student satisfaction and learning outcomes.</p><p><strong>Results: </strong>A total of 39 students participated in the study. The mean satisfaction score of the intervention group was significantly higher than that of the control group (P=0.027). The mean score of learning outcomes was also significantly higher for the intervention than the control group (P=0.028).</p><p><strong>Conclusion: </strong>Smartphone technology can provide a platform for improving the evaluation of the clinical skills of nursing anesthesia students, leading to increased satisfaction and improved learning outcomes.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"20 ","pages":"10"},"PeriodicalIF":4.4,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10169697/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9449274","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Experience of introducing an electronic health records station in an objective structured clinical examination to evaluate medical students’ communication skills in Canada: a descriptive study.","authors":"Kuan-Chin Jean Chen, Ilona Bartman, Debra Pugh, David Topps, Isabelle Desjardins, Melissa Forgie, Douglas Archibald","doi":"10.3352/jeehp.2023.20.22","DOIUrl":"10.3352/jeehp.2023.20.22","url":null,"abstract":"<p><strong>Purpose: </strong>There is limited literature related to the assessment of electronic medical record (EMR)-related competencies. To address this gap, this study explored the feasibility of an EMR objective structured clinical examination (OSCE) station to evaluate medical students’ communication skills by psychometric analyses and standardized patients’ (SPs) perspectives on EMR use in an OSCE.</p><p><strong>Methods: </strong>An OSCE station that incorporated the use of an EMR was developed and pilot-tested in March 2020. Students’ communication skills were assessed by SPs and physician examiners. Students’ scores were compared between the EMR station and 9 other stations. A psychometric analysis, including item total correlation, was done. SPs participated in a post-OSCE focus group to discuss their perception of EMRs’ effect on communication.</p><p><strong>Results: </strong>Ninety-nine 3rd-year medical students participated in a 10-station OSCE that included the use of the EMR station. The EMR station had an acceptable item total correlation (0.217). Students who leveraged graphical displays in counseling received higher OSCE station scores from the SPs (P=0.041). The thematic analysis of SPs’ perceptions of students’ EMR use from the focus group revealed the following domains of themes: technology, communication, case design, ownership of health information, and timing of EMR usage.</p><p><strong>Conclusion: </strong>This study demonstrated the feasibility of incorporating EMR in assessing learner communication skills in an OSCE. The EMR station had acceptable psychometric characteristics. Some medical students were able to efficiently use the EMRs as an aid in patient counseling. Teaching students how to be patient-centered even in the presence of technology may promote engagement.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"20 ","pages":"22"},"PeriodicalIF":4.4,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10442496/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10055182","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Effect of motion-graphic video-based training on the performance of operating room nurse students in cataract surgery in Iran: a randomized controlled study","authors":"Behnaz Fatahi, Samira Fatahi, Sohrab Nosrati, Masood Bagheri","doi":"10.3352/jeehp.2023.20.34","DOIUrl":"10.3352/jeehp.2023.20.34","url":null,"abstract":"<p><strong>Purpose: </strong>The present study was conducted to determine the effect of motion-graphic video-based training on the performance of operating room nurse students in cataract surgery using phacoemulsification at Kermanshah University of Medical Sciences in Iran.</p><p><strong>Methods: </strong>This was a randomized controlled study conducted among 36 students training to become operating room nurses. The control group only received routine training, and the intervention group received motion-graphic video-based training on the scrub nurse’s performance in cataract surgery in addition to the educator’s training. The performance of the students in both groups as scrub nurses was measured through a researcher-made checklist in a pre-test and a post-test.</p><p><strong>Results: </strong>The mean scores for performance in the pre-test and post-test were 17.83 and 26.44 in the control group and 18.33 and 50.94 in the intervention group, respectively, and a significant difference was identified between the mean scores of the pre- and post-test in both groups (P=0.001). The intervention also led to a significant increase in the mean performance score in the intervention group compared to the control group (P=0.001).</p><p><strong>Conclusion: </strong>Considering the significant difference in the performance score of the intervention group compared to the control group, motion-graphic video-based training had a positive effect on the performance of operating room nurse students, and such training can be used to improve clinical training.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"20 ","pages":"34"},"PeriodicalIF":4.4,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11009010/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138446569","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Editorial policies of Journal of Educational Evaluation for Health Professions on the use of generative artificial intelligence in article writing and peer review","authors":"Sun Huh","doi":"10.3352/jeehp.2023.20.40","DOIUrl":"10.3352/jeehp.2023.20.40","url":null,"abstract":"","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"20 ","pages":"40"},"PeriodicalIF":9.3,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11893185/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139058879","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Information amount, accuracy, and relevance of generative artificial intelligence platforms’ answers regarding learning objectives of medical arthropodology evaluated in English and Korean queries in December 2023: a descriptive study","authors":"Hyunju Lee, Soobin Park","doi":"10.3352/jeehp.2023.20.39","DOIUrl":"10.3352/jeehp.2023.20.39","url":null,"abstract":"<p><strong>Purpose: </strong>This study assessed the performance of 6 generative artificial intelligence (AI) platforms on the learning objectives of medical arthropodology in a parasitology class in Korea. We examined the AI platforms’ performance by querying in Korean and English to determine their information amount, accuracy, and relevance in prompts in both languages.</p><p><strong>Methods: </strong>From December 15 to 17, 2023, 6 generative AI platforms—Bard, Bing, Claude, Clova X, GPT-4, and Wrtn—were tested on 7 medical arthropodology learning objectives in English and Korean. Clova X and Wrtn are platforms from Korean companies. Responses were evaluated using specific criteria for the English and Korean queries.</p><p><strong>Results: </strong>Bard had abundant information but was fourth in accuracy and relevance. GPT-4, with high information content, ranked first in accuracy and relevance. Clova X was 4th in amount but 2nd in accuracy and relevance. Bing provided less information, with moderate accuracy and relevance. Wrtn’s answers were short, with average accuracy and relevance. Claude AI had reasonable information, but lower accuracy and relevance. The responses in English were superior in all aspects. Clova X was notably optimized for Korean, leading in relevance.</p><p><strong>Conclusion: </strong>In a study of 6 generative AI platforms applied to medical arthropodology, GPT-4 excelled overall, while Clova X, a Korea-based AI product, achieved 100% relevance in Korean queries, the highest among its peers. Utilizing these AI platforms in classrooms improved the authors’ self-efficacy and interest in the subject, offering a positive experience of interacting with generative AI platforms to question and receive information.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"20 ","pages":"39"},"PeriodicalIF":9.3,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11893187/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139049470","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Negative effects on medical students' scores for clinical performance during the COVID-19 pandemic in Taiwan: a comparative study.","authors":"Eunice Jia-Shiow Yuan, Shiau-Shian Huang, Chia-An Hsu, Jiing-Feng Lirng, Tzu-Hao Li, Chia-Chang Huang, Ying-Ying Yang, Chung-Pin Li, Chen-Huan Chen","doi":"10.3352/jeehp.2023.20.37","DOIUrl":"10.3352/jeehp.2023.20.37","url":null,"abstract":"<p><strong>Purpose: </strong>Coronavirus disease 2019 (COVID-19) has heavily impacted medical clinical education in Taiwan. Medical curricula have been altered to minimize exposure and limit transmission. This study investigated the effect of COVID-19 on Taiwanese medical students' clinical performance using online standardized evaluation systems and explored the factors influencing medical education during the pandemic.</p><p><strong>Methods: </strong>Medical students were scored from 0 to 100 based on their clinical performance from 1/1/2018 to 6/31/2021. The students were placed into pre-COVID-19 (before 2/1/2020) and midst-COVID-19 (on and after 2/1/2020) groups. Each group was further categorized into COVID-19-affected specialties (pulmonary, infectious, and emergency medicine) and other specialties. Generalized estimating equations (GEEs) were used to compare and examine the effects of relevant variables on student performance.</p><p><strong>Results: </strong>In total, 16,944 clinical scores were obtained for COVID-19-affected specialties and other specialties. For the COVID-19-affected specialties, the midst-COVID-19 score (88.513.52) was significantly lower than the pre-COVID-19 score (90.143.55) (P<0.0001). For the other specialties, the midst-COVID-19 score (88.323.68) was also significantly lower than the pre-COVID-19 score (90.063.58) (P<0.0001). There were 1,322 students (837 males and 485 females). Male students had significantly lower scores than female students (89.333.68 vs. 89.993.66, P=0.0017). GEE analysis revealed that the COVID-19 pandemic (unstandardized beta coefficient=-1.99, standard error [SE]=0.13, P<0.0001), COVID-19-affected specialties (B=0.26, SE=0.11, P=0.0184), female students (B=1.10, SE=0.20, P<0.0001), and female attending physicians (B=-0.19, SE=0.08, P=0.0145) were independently associated with students' scores.</p><p><strong>Conclusion: </strong>COVID-19 negatively impacted medical students' clinical performance, regardless of their specialty. Female students outperformed male students, irrespective of the pandemic.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"20 ","pages":"37"},"PeriodicalIF":4.4,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10810719/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139040715","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}