From Assistance to Autonomy: Evaluating Procedural Competency in Pediatric Emergency Medicine

Abstract

Objective

Pediatric emergency medicine (PEM) physicians require expertise in numerous procedural skills to manage emergencies in children. Fellows require hands-on experience, expert supervision, and standardized feedback to build procedural competency. Our objective was to develop and gather validity evidence for an evaluation tool to assess PEM fellows' ability to perform procedures.

Methods

We conducted a retrospective study of PEM fellows' procedural performance within a children's hospital system. Faculty evaluated fellows using a one-item, five-point entrustment/level of supervision scale. We focused on three frequently performed procedures: laceration repair, intubation, and procedural sedation. To assess changes in supervision scores by training year, we used a mixed-effects binary logistic regression model with scores ≥ 4 as the threshold. Adjusted odds ratios (aOR), 95% confidence intervals (CIs) and p values were reported. We also assessed inter- and intra-rater reliability overall, by procedure, and by semester.

Results

Data from 38 fellows and 137 supervising faculty evaluations were included. Compared to first-year fellows, second-year fellows were significantly more likely to receive higher scores (aOR = 4.95; 95% CI 3.78–6.47) and third-year fellows even more so (aOR = 10.15; 95% CI 6.71–15.35). Intra-rater reliability showed moderate to very strong correlation (ρ = 0.84), and by procedure: intubation (ρ = 0.78), lac repair (ρ = 0.92), and sedation (ρ = 0.85). Inter-rater reliability was poor across all measures. First-year fellows showed significant differences in intubation scores between specialties. No significant differences were found among third-year fellows.

Conclusion

A supervision-based evaluation tool demonstrated strong intra-rater reliability and captured increasing procedural competency among PEM fellows, as evidenced by progressively higher entrustment scores with increasing years in training. This tool may support standardized assessment and meaningful feedback throughout fellowship training.