Validity evidence of a resuscitation team leadership assessment measure for use in actual trauma resuscitations

Abstract

Background

Team leadership is a critical skill in trauma resuscitation teams, linked to better teamwork and improved patient care. There are numerous published team leadership assessments, though data regarding the performance of these measures in patient care settings (vs. simulation-based settings) remain limited. There remains a need for a valid, reliable, and efficient measure of resuscitation team leadership in the clinical setting to support medical education and research efforts.

Methods

We constructed a 12-item behaviorally anchored rating scale (BARS) to measure trauma team leadership. Multiple raters then used the BARS to measure team leadership in 360 recorded trauma resuscitations across 60 participants. In addition to examining inter-rater reliability, we examined the construct validity of the BARS assessment through both correlational and latent modeling techniques to compare the ratings collected with the BARS to those collected using a previously studied checklist-based assessment using a multitrait–multimethod (MTMM) approach. Lastly, we examined the criterion validity of the BARS measure by examining its relationship with previously obtained patient care scores.

Results

BARS items demonstrated high inter-rater reliability when scores were computed using observations averaged over multiple raters (mean item intraclass correlations ICC1k 0.90, item range 0.85–0.98). The correlation between the aggregate ratings from the team leadership BARS and checklist measure demonstrated a strong positive correlation (r = 0.75), and the MTMM analyses indicated consistent evidence for both convergent (mean monotrait–heteromethod r = 0.50) and discriminant (mean heterotrait–heteromethod r = 0.27) validity. Hierarchical Bayesian regression analyses revealed that aggregate BARS scores were predictive of patient care scores (β = 7.06, 95% HDI 3.76–10.43).

Conclusions

The team leadership BARS and a previously studied checklist-based team leadership measure produced convergent assessments of team leadership behavior in the present data. Furthermore, higher overall ratings on the BARS correlated with better patient care delivery at the team level.