Item-Level Scoring Profiles and Technical Domains in the Colorectal Endoscopic Surgical Skill Qualification Examination

Abstract

Background

Video-based assessment is increasingly used to evaluate operative performance, but pass–fail outcomes alone provide limited information on the technical components associated with the decision. This study aimed to clarify item-level scoring patterns and latent technical domains underlying pass–fail decisions in the colorectal Endoscopic Surgical Skill Qualification examination.

Methods

This secondary item-level analysis used the colorectal ESSQS examination dataset of the Japan Society for Endoscopic Surgery from 2020 to 2024. The final application-level pass–fail decision was the primary outcome, and each primary reviewer evaluation was the unit of analysis. Item scores were standardized as proportions of the version-specific maximum. Univariable logistic regression, multivariable generalized linear mixed-effects models, and exploratory factor analysis were performed.

Results

A total of 1945 applications yielded 3890 reviewer-level observations, with a final pass rate of 33.2%. Items related to operative field exposure, cooperation with assistants, dissection-plane recognition, workflow planning, vascular control, and transection/anastomosis were most strongly associated with the outcome. Exploratory factor analysis identified seven clinically interpretable domains; operative field exposure and team coordination, dissection-plane recognition, vessel management and lymphadenectomy, and transection/anastomosis were most strongly associated with the final decision.

Conclusions

Pass–fail decisions in the colorectal ESSQS examination were characterized by both general endoscopic competencies and colorectal procedure-specific technical components. Item- and domain-level analyses describe the internal structure of the rubric; application to individualized feedback would require improved inter-reviewer reliability and further validation, and the findings may inform reviewer calibration and refinement of video-based skill assessment.