Enhancing OSCE reliability and effectiveness in radiology resident training with long-term systemic evaluation.

Objective: This study aimed to evaluate the long-term systematic effectiveness and reliability of the Objective Structured Clinical Examination (OSCE) in radiology resident training, from the perspectives of both examiners and examinees.

Methods: This retrospective observational study analyzed subjective evaluations and objective examination data collected over 6 years (2018-2021, 2023, and 2024). Subjective evaluations were gathered via questionnaires from 198 examiners and 818 examinees to assess the difficulty and satisfaction with the OSCE. Objective data, including examination scores, difficulty indices, and discrimination indices, for each OSCE station were analyzed using correlation analysis and t-tests.

Results: The OSCE demonstrated stable performance over 6 years, with consistent difficulty levels and discrimination ability across all stations. The average scores for individual stations varied; however, the overall final scores remained stable. Strong correlations between the station and final scores indicate good discrimination. Examinees rated the overall difficulty higher than examiners, but the objective indices aligned with examiner assessments. Over 6 years (198 examiners, 818 examinees), OSCE scores stabilized (85.48-88.48), with improved consistency (station range narrowed to 85.51-93.9 by 2024). Difficulty (0.12-0.15) and discrimination indices remained stable (most p < 0.05). Examinees rated it harder than examiners (p < 0.001).

Conclusion: The OSCE is a reliable, valid, and effective assessment tool in radiology. Evaluating the OSCE from both subjective and objective perspectives ensured the robustness and validity of the examination.

Critical relevance statement: This 6-year study evaluates the Objective Structured Clinical Examination (OSCE) in radiology training through multidimensional analysis of examination metrics (difficulty indices and discrimination coefficients) and stakeholder feedback (n = 198 examiners, 818 examinees), demonstrating its consistency for clinical competency assessment.

Key points: The radiology OSCE demonstrated consistent reliability, stable difficulty indices, and strong score correlations. Examinees overestimated exam difficulty compared to examiners, likely due to stress-related perception bias. Standardized examiner training improved scoring consistency and enhanced overall OSCE assessment quality.