Examining the Reliability and Validity of the ALS Certification Examinations with the Inclusion of Clinical Judgment: An Update on the ALS Examination Redesign.

Objectives: Clinical judgment describes the process an emergency medical service clinician uses to evaluate problems and make decisions in the out-of-hospital setting. As part of the redesign of the Advanced Life Support (ALS) certification examinations, the National Registry of Emergency Medical Technicians is developing and evaluating items that measure clinical judgment, with the intention of assessing these as a new domain in the ALS certification examinations. In this study, we provide evidence around the redesign by evaluating the reliability and validity of the advanced emergency medical technician (AEMT) and paramedic certification examinations when clinical judgment is included as a sixth domain along with the five current domains.

Methods: Pretest (i.e., pilot, unscored) clinical judgment items were included as a new sixth clinical judgment domain. We then used the combination of operational (i.e., scored) and pretest items for all six domains and scored the redesigned AEMT and paramedic certification examinations. We evaluated the psychometric properties of these ALS examinations within the Rasch measurement framework with multiple assessments of reliability and validity including item-level statistics (e.g., mean-square infit and outfit, local dependence) and examination-level statistics (e.g., person reliability, item reliability, item separation, decision consistency, decision accuracy). Wright Maps were produced to evaluate whether the examination item difficulty statistics aligned with the candidate ability continuum.

Results: The total population of all examination forms included were 20,136 (AEMT 4,983; paramedic 15,153). The Rasch-based statistics for the redesigned AEMT and paramedic examinations, for both item and examination-level statistics, were well within the psychometric standard values. Wright maps demonstrated that the developed items fall along the candidate ability continuum for both examinations. Further, the distribution of clinical judgment item difficulties fell within the current item distribution, providing evidence that these new items are of similar difficulty to the items measuring the five current domains.

Conclusion: We demonstrate strong reliability and validity evidence to support that the integrity of the examinations is upheld with the addition of clinical judgment items, while also providing a more robust candidate evaluation. Most importantly, the pass/fail decisions that candidates receive accurately reflect their level of ALS knowledge at the entry-level.