Automated quality assurance for emergency department documentation: pilot comparison with physician peer review of simulated chest pain cases.

Abstract

Objectives: To evaluate the feasibility and reliability of an artificial intelligence-driven quality assurance system for emergency chest pain documentation in simulated cases, compared to traditional physician reviewers.

Methods: We developed an automated quality assurance solution leveraging commercial Large Language Models that are customized based on College of Physicians and Surgeons of Ontario peer review standards. Seventeen simulated emergency chest pain cases of varying quality were independently reviewed by an automated auditor, six community emergency physicians, and original case writers using a 3-point evaluation scale across nine dimensions. Agreement was measured using Lin's Concordance Correlation Coefficient for overall score agreement, Root Mean Square Error for overall score accuracy, and Cohen's Kappa statistics for categorical dimension scores.

Results: Inter-rater agreement among human reviewers showed substantial variability with overall score concordances ranging from 0.28 to 0.86. The automated auditor demonstrated agreement levels within the range of human variability within the limited sample of simulated cases with the best performing model (guideline-customized Claude Sonnet 4) with a concordance value of 0.85 and root mean square error of 0.25 with respect to the average human scores on the 3-point scale. For pass/fail decisions, agreement with human reviewers ranged from 65 to 94% with Kappa scores of 0.27 to 0.85, comparable to inter-human reviewer agreement Kappa scores of - 0.12 to 0.68.

Conclusions: The performance of the automated auditor showed agreement within the range of variability observed among that of the small group of uncalibrated human reviewers for the simulated cases in determining documentation quality and pass/fail outcomes. Given the potential variability in human assessments, automated quality assurance may provide a more consistent evaluation of quality, with the added advantage of the mass-processing of many cases in near real-time.