Validity evidence of the 2022 ERS thoracic ultrasound Objective Structured Clinical Examination (OSCE)

Abstract

Background: Clinicians increasingly use thoracic ultrasound in several settings due to its high diagnostic accuracy for many common causes of respiratory failure and dyspnoea. However, ultrasound examinations are operator-dependent, and sufficient competences are needed to obtain high diagnostic accuracy and to attain the ability to interpret the images and integrate them into the patient history and other examinations. This study aims to explore and gather validity evidence for the 2022 ERS thoracic ultrasound objective structured clinical examination (OSCE). Methods: An expert panel created the test, which included two theoretical and three practical stations comprising cases with diagnoses which can be established by point-of-care thoracic ultrasound. Twenty-five participants with different levels of experience thoracic ultrasound completed the test. Data from the test scores and questionnaires on participants’ experiences were used for item analysis. Validation was done according to Messick’s framework. The contrasting groups9 standard-setting method was used to establish a pass/fail score. Results: The summarised internal consistency reliability was high with a Cronbach’s alpha=0.83. The novice group (n=4) had a mean test score of 26.9±10.0 points, the intermediate group (n=8) scored 55.2±5.8 points, and the experienced group (n=13) 62.0±6.1 points (one-way ANOVA, p<0.001). A pass/fail score of 48 points was thus derived (maximum score =75 points). Conslusion: We developed a test for the assessment thoracic ultrasound competences with solid validity evidence, and a pass/fail standard with no false positives or false negatives.