"But Why?": Explanatory Feedback Is a Reliable Marker of High-Quality Narrative Assessment of Surgical Performance.

Abstract

Purpose: This study examines the quality of short narrative comments collected using an online workplace-based assessment (WBA) tool.

Method: The quality of comments collected by a WBA tool at the UPMC Ophthalmology Residency Training Program was evaluated between July 2017-June 2020. A randomized rating exercise involving 10 meta-raters from 6 institutions was performed to evaluate the value of narratives from deidentified WBAs. The tool captured a single-item entrustment competency question with brief comments. Comments were evaluated using a Quality of Assessment of Learning (QuAL) score (range, 0-5; ≥ 3 considered high quality) and on whether the assessor provided a feedback rationale.

Results: A total of 838 unique WBAs were collected from 15 attending evaluators. Comments were brief (median [interquartile range] length, 11 [7-17] words), yet 514 (61.3%) were rated as high quality (QuAL score ≥ 3). Of all 838 comments, 98 (11.7%) included a specific reason the evidence or suggestion was provided to the learner. Of these 98 comments, 94 (95.9%) met the high-quality feedback threshold. A higher QuAL score was associated with a higher postgraduate year (PGY) level (estimate [SE], 1.603 [0.428], P < .001 for PGY2 [reference]; 1.003 [0.389], P = .01 for PGY3; 1.079 [0.360], P = .003 for PGY4), suggesting more advanced learners receive higher-quality narrative comments. A correlation was found between a higher entrustment rating and a lower QuAL score (estimate [SE], -0.199 [0.053], P < .001). When the PGY level was controlled for, this association got stronger (estimate [SE], -0.310 [0.057], P < .001).

Conclusions: Analysis of WBA comments from attending physicians evaluated using the QuAL score demonstrated that most comments were high quality despite their brevity. Residents in later training years and with lower entrustment ratings received higher-quality comments. High-quality narrative assessments were longer and addressed rationale as part of the comment.