Developing the Productive Academic Formulas Test: Evidence from Rasch and factor analysis and cognitive fluency analysis

Abstract

The aim of this study is to explore the validity of the Productive Academic Formulas Test (PAFT), developed from items in the Academic Formulas List (Simpson-Vlach & Ellis, 2010). A total of 110 Japanese undergraduate students learning English took the PAFT, the Productive Vocabulary Levels Test (PVLT; Laufer & Nation, 1999), and the Phrasal Vocabulary Size Test (PVST; Martinez, 2011). After calibrating the PAFT items using Rasch dichotomous modeling, the convergent and discriminant validity of the PAFT was examined through confirmatory factor analysis (CFA). The four theoretical CFA models were tested: (1) single-factor model (vocabulary knowledge), (2) two-factor model (single vocabulary vs. multiword sequence), (3) two-factor model (receptive vocabulary vs. productive vocabulary), and (4) three-factor model (single vocabulary vs. productive multiword sequence vs. receptive multiword sequence). The predictive power of the PAFT was also examined by predicting L2 utterance fluency measures (speed, breakdown, and repair fluency) elicited from an argumentative speech task. The results indicated that the PAFT demonstrated satisfactory psychometric properties (unidimensionality and high reliability) and construct validity as a measure of productive multiword sequence knowledge. Regression analyses with dominance analysis procedure revealed that the PVLT scores contributed most strongly to most of the fluency measures (dominance weight: 34.9 %–59.1 %), followed by the scores of PAFT (20.1 %–30.2 %) and PVST (13.2 %–45.1 %). Implications for vocabulary testing and for the contribution of vocabulary knowledge to fluency are discussed.