Evaluation of the information content of individual items' scores in the anesthesiologist supervision instrument using Cochran's Q tests and McNemar's tests to provide specific feedback to ratees in addition to reliable evaluation of clinical performance

Background

Annual professional practice evaluations (i.e., peer review) are mandatory for anesthesiologists in many practice settings. The de Oliveira Filho clinical anesthesia supervision instrument is a valid and psychometrically reliable tool suitable for these high-stakes assessments. We studied item-specific feedback to anesthesiologists to increase their scores.

Methods

The retrospective cohort study used all 11 academic years for which the studied department used the supervision instrument, July 2013 through June 2024. There were 55,195 evaluations of 715 combinations of anesthesiologist and year by 242 rating trainees (e.g., anesthesia residents), each evaluation with 9-items scored 4=always, 3=frequently, 2=rarely, or 1=never.

Results

The 9-item supervision instrument had Cronbach alpha 0.96 and functioned as a binary (4 vs ≤3) multivariate array. Consequently, the 87% (47,859/55,195) of evaluations with all 9 items 4=always, or all 9 items ≤3, provided information about the quality of performance of the anesthesiologists, but no potentially useful item-specific information for feedback to anesthesiologists. Cochran Q tests were performed for each of the 715 combinations of anesthesiologist and year using the remaining 7336 evaluations. There were 17% (124/715) of the combinations of anesthesiologist and year with adjusted P < 0.05, showing one or more of the items' scores differed significantly from the other items' scores. The 17% of combinations represented 6.0% (3311/55,195) of evaluations. For each of those 124 combinations of anesthesiologist and year, 36 McNemar tests were performed, comparing the 1st item to the 2nd, …, 8th item to 9th. Among those pairwise comparisons that were statistically significant, the directions of odds ratios were examined. The items about teaching quality accounted for 19% and 26% of the odds ratios <1 (i.e., low scores), respectively, while the other seven items each accounted for ≤5%.

Conclusions

Earlier it was known that anesthesiologists' annual professional practice evaluations can be provided along with education regarding good teaching attributes associated with high quality intraoperative supervision and greater supervision scores. Our results show that also providing analyses of individual item scores could benefit, at most, <20% of the faculty anesthesiologists.