A meta-analysis of the reliability of a metacognitive awareness instrument in second language listening

Metacognitive awareness is essential in regulating second language (L2) listening and has been predominantly assessed by a multidimensional instrument named the Metacognitive Awareness Listening Questionnaire (MALQ). Since previous studies have yielded inconclusive evidence concerning the generalization of MALQ, it is important to examine the overall reliability of the MALQ measures from a meta-analytical perspective. The purpose of the study was to examine variability in the reliability of MALQ measures in the field of L2 listening. A meta-analytic reliability generalization (RG) was conducted to synthesize Cronbach’s alpha coefficients derived from 45 studies that used MALQ. The results showed that the aggregated reliability estimate was 0.80 for MALQ measures, with four out of the five subscales having an aggregate reliability coefficient larger than 0.7, i.e., 0.73 for mental translation, 0.74 for planning and evaluating, 0.71 for person knowledge, and 0.79 for problem-solving. On the other hand, the reliability of directed attention was 0.68, falling short of meeting the minimum requirement of 0.70. In addition, as a high degree of heterogeneity was found in the studies included, a mixed effect meta-regression was performed, identifying four moderators affecting the reliability of MALQ measures: publication year, educational setting, participants’ L1, and L2 proficiency level. We further found evidence for publication bias in the included publications. Suggestions for future research are provided.