Evaluating AI literacy of secondary students: Framework and scale development

K-12 AI education not only equips students with AI literacy but also encourages underrepresented groups to pursue further studies or careers in this field. Secondary students were particularly well-suited for comprehensive AI education due to their cognitive characteristics and developmental readiness. While most studies have focused on developing pedagogy, curriculum, and tools for secondary AI education, they have prioritized measuring students' learning outcomes over literacy development. Referring to the empirical research on secondary AI education as well as Piaget's Epistemology and Bloom's Taxonomy, this study figured out a KAT framework that constitutes AI literacy: (1) AI Knowledge (AI fundamentals, elements of AI technology, application of AI technology), (2) AI Affectivity (AI and human, AI and society), and (3) AI Thinking (engineering design thinking, computational thinking). Based on this, a 57-item AI literacy scale (AILS) was developed, and 56 items were retained after expert judgement. Then, a large sample of Chinese secondary students was surveyed, resulting in 1392 valid samples, which were randomly divided into two sub-samples: 720 samples were used for item reduction through Rasch Analysis and Exploratory Factor Analysis; 672 samples were used for model validation and comparison through Confirmatory Factor Analysis. Results indicated the AILS with three-factor structure of 48 items has a good reliability and validity. Moreover, gender differences in AI literacy among secondary students were examined. Results indicated that boys had significantly higher AI Knowledge than girls, whereas girls had significantly higher AI Affectivity than boys. The implications, limitations and future research were also discussed.