Assessing interactional competence through group discussion: A mixed methods validation

Abstract

The purpose of this mixed methods study was to assess the validity and functionality of an analytic rating scale for the assessment of interactional competence (IC). The participants were 79 low- to high-proficiency Japanese university students who completed 10-minute small-group discussions. Video recordings of the discussions were assessed by raters using the rating scale. The rater scores were then analyzed using many-facet Rasch measurement (MFRM) which indicated a very good fit to the model. The data were subsequently analyzed using generalizability theory in the form of a G-study and d-study. Those studies showed that the rating scale could be used with fewer raters, therefore increasing practicality without a substantial decrease in reliability. In addition to quantitative data, qualitative data were also collected in the form of interviews with raters and comments they made during assessment. Several raters noted unexpected participant behaviors which were difficult to evaluate using the rating scale, as well as ambiguous language in some category descriptors. The qualitative data provided an invaluable supplement to the quantitative analyses which did not indicate major issues with the rubric. Both forms of data were used to revise the original rating scale and those changes are discussed. This study adds to the limited but growing number of mixed methods studies on IC assessment.