Validity evidence for a team-leading assessment tool in pediatric emergency resuscitations using video review

Abstract

Background

Effective leadership of health care action teams has demonstrated positive influence on team performance and patient care, but there is no consensus on how to assess these skills. We developed a novel team leadership assessment tool for leaders of interprofessional pediatric resuscitation teams and collected validity evidence for this tool using video review.

Methods

This was a prospective cohort study from November 2021 to October 2022. A novel team leadership assessment tool was developed using literature review and local expertise and then piloted and refined using medical simulation. Pediatric emergency medicine (PEM) fellows from a single tertiary care pediatric medical center were enrolled, and videos of one medical resuscitation and one trauma resuscitation were collected per fellow each month. Three reviewers underwent reviewer training and then scored the videos using the assessment tool. Raters provided feedback on feasibility and ease of use using a 5-point Likert scale. Inter-rater reliability for the assessment tool using Gwet's agreement coefficient and the association between performance and clinical level of training using generalized linear mixed model were calculated.

Results

Twelve PEM fellows enrolled and 146 videos were reviewed. The inter-rater reliability for each domain ranged from 0.45 (p < 0.0001) to 0.59 (p < 0.0001), with the inter-rater reliability of the total score being 0.49 (p < 0.0001). The reviewers’ mean ratings of the elements of the tool were as follows: clarity of the domains (4.6/5), the independence of each domain from each other (3.9/5), the ease of use of the 5-point Likert scale (4.5/5), the usefulness of the provided examples for each domain (4.6/5), and the ability to assess each domain without having to rewatch (4.5/5). The tool differentiated between levels of clinical training for two of the six domains (p < 0.02).

Conclusions

We developed a novel team leadership assessment tool for pediatric resuscitation team leaders that demonstrated moderate inter-rater reliability. The tool was easy to use and feasible for educators to assess the performance of PEM trainees in complex high-stakes clinical situations.