Development and validation of a test to assess teachers' knowledge of how to operate technology

Abstract

To effectively adopt technology during teaching, teachers require knowledge of how to operate technology. Especially first-time technology users need knowledge of how to handle digital devices and software programs as a foundation to use technology in the classroom successfully. This knowledge has so far been assessed mainly using self-reports. However, self-assessments are insufficient for assessing knowledge as their validity is limited. Moreover, the few tests that exist to measure technological knowledge (TK) show weaknesses (e.g., lack of ecological validity, outdated items). We present a test assessing teachers' TK that is independent of specific operating systems, covers technology that is relevant in everyday teaching, and is grounded in acknowledged psychometric modeling principles. We iteratively developed a test (named T-TK) comprising 26 items, utilizing cognitive testing, expert feedback, and two studies (N_{pilot study} = 268 pre-service and in-service teachers, N_{main study} = 233 in-service teachers) to filter items that did not match in content and were not Rasch conform. T-TK showed a satisfactory Andrich's reliability (Rel_Andrich = 0.73). Using the sample N_{main study}, correlations between T-TK and technological knowledge (self-report, r = 0.52), pedagogical knowledge (test scores, r = 0.18), and technological pedagogical knowledge (self-report, r = 0.33; test scores, r = 0.46) indicated convergent and discriminant validity. Thus, the T-TK proves to be a reliable and valid instrument to capture teachers' TK. The T-TK can be used both by practitioners not requiring any statistical knowledge (e.g., for individual diagnostics) and in research (e.g., to analyze teachers' TK).