Assessing fundamental clinical skills of osteopathic medical students.

Abstract

Context: Clinically based performance assessments, including those employing standardized patients (SPs), have been utilized to educate and assess medical students and graduates. Starting in the 1990s, demonstrating competency on these types of assessments became part of the medical licensure pathway in some countries, and was re-added to the licensure pathway in the United States in 2004. Studies have been published to support the validity of the scores and associated pass/fail decisions obtained from SP assessments. With the onset of the COVID-19 pandemic, testing organizations were forced to suspend many of their in-person clinical skills assessments. There is widespread recognition that clinical skills are necessary for high-quality patient care. Nevertheless, there has been disparate interest in reestablishing national assessments of these skills.

Objectives: Based on the recommendations of the Special Commission on Osteopathic Medical Licensure Assessment, the National Board of Osteopathic Medical Examiners (NBOME) established the Core Competency Capstone for Doctors of Osteopathic Medicine (C3DO). The objectives of this study were to describe the assessment and to ascertain the feasibility of administering it at multiple colleges of osteopathic medicine (COMs).

Methods: Surveys of osteopathic medical school personnel and students taking the assessment were conducted to gather data on the feasibility of administering the C3DO, the challenges associated with a distributed model, and the educational value of the assessment. Reports submitted by the medical schools provided information on the cost of the assessment. Some psychometrics analyses of Phase 1 C3DO pilot data were conducted, including the quantification of measurement errors.

Results: The NBOME, in collaboration of the COMs, was able to administer the C3DO to almost the entire class at four osteopathic medical schools. Although there were challenges, including data transfer from the schools, summary scores were derived for a total of 811 students, including nine repeat test takers. Medical students across sites indicated that the C3DO was a valuable educational assessment activity. The reliability of the ratings derived from the checklists (history building [HB], physical examination [PE]) and Communication and Respectfulness Evaluation (CARE) tool were adequate. Based on a variance components analysis, there was some evidence to suggest that at a given school, the choice of an SP portraying and scoring the case could have an impact on the student scores.

Conclusions: The C3DO-distributed site model, which includes centrally developed cases, standardized training of SPs, and independent scoring of osteopathic manipulative treatment (OMT), provides the means to assess the clinical skills of osteoapthic medical students at different testing sites. The implementation of a distributed, school-based model for the assessment of osteopathic clinical skills is feasible, potentially economically advantageous, and if rigorously developed and administered, should provide for comparable student scores and associated competency decisions. Additional Phase 2 (2024-2025) and Phase 3 (2025-2026) studies are underway to establish the psychometric rigor of the assessment more fully.