Development of the trainer skill evaluation scale for metacognitive training.

Aim: The purpose of this study is manifold: to develop a trainer skill rating scale for metacognitive training (MCT), to determine the difficulty level of the behavioral checklist, and to examine the reliability and validity of the MCT Trainer Skills Rating Scale.

Method: In Study 1, an MCT trainer skill behavior checklist was developed with expert staff members, and a questionnaire was administered to MCT trainers. Item categorization was identical to that used in previous studies. In Study 2, a video was used to conduct the survey. All subjects were given a 1-hour training session, instructed on evaluating the MCT Trainer Skills Rating Scale, and asked to rate their trainer skills on a mock video designed for beginners and a mock video designed for advanced trainers.

Result: In Study 1, responses from 49 respondents were obtained. The survey results showed that 72 items were classified similarly to previous studies. In Study 2, two pairs were randomly selected, and weighted kappa coefficients were calculated for the sub-items of the MCT Trainer Skills Rating Scale. High agreement was obtained with K = 0.71 and K = 0.73, indicating high reliability.

Conclusion: High reliability was obtained for all eight items of the MCT Trainer Skills Rating Scale created in this study. In addition, the video evaluation scores for the advanced trainer were significantly higher than those for the beginner trainer, suggesting that discriminant validity was confirmed among the criterion-related validity. These results confirm that the scale has both high reliability and validity.