Investigating the Skills Involved in Reading Test Tasks through Expert Judgement and Verbal Protocol Analysis: Convergence and Divergence between the Two Methods

ABSTRACT Expert judgement has been frequently employed with reading assessments to gauge the skills potentially measured by test tasks, for purposes such as construct validation or producing diagnostic information. Despite the critical role it plays in such endeavours, few studies have triangulated its results with other types of data such as reported test-taking processes. A lack of such triangulation may bring the validity of experts’ judgements into question and undermine the credibility of subsequent procedures that build on them. In light of this, this study compared two groups of language experts’ judgements on the content of two sets of reading test tasks with ten university students’ verbal reports on solving those tasks. It was found that convergence was achieved between the two information sources for about 53% of the test tasks on what they were mainly assessing. However, there was a bigger gap between them regarding the specific skills involved in each task. A careful examination of the discrepancies between the two sources revealed that they are attributable to a number of factors. This study highlights the need to cross-check the results of expert judgement with other data sources. Implications for future test development and research are also discussed.