Program Evaluation and Triangulation: What Three Data Sets Show about How Participants Respond to the Course Teaching Academic English with the TOEFL iBT® Test

Abstract

Drawing from a larger CIPP program evaluation (Context, Input, Processes, Products) comprised of 11 analyses of the ETS online teacher training course titled Teaching Academic English with the TOEFL iBT® Test (TAE), this paper triangulates three qualitatively different data sets and seeks to understand how participants responded to the TAE course. The research team considered self-reported data (closed- and open-answer survey questions), engagement data, and learning data. The self-reported data were strongly positive and illustrated the value participants reported having about the course content, especially ETS-created materials. Some gaps were indicated: calls were made for more materials—especially additional resources participants could use when teaching—and for increased interactions during the course workshop. In partial juxtaposition, the engagement data indicated that participants were engaging the course less than anticipated. The learning data simultaneously indicated that some assessments were quite easy for participants while elsewhere participants did not have the desired uptake of some topics. Viewing the three data sets together, the research team arrived at a more nuanced conclusion: despite participant satisfaction, development processes should continue, to ensure the highest quality teacher training program possible, in terms of its content, resources, and interactions.