Development of a generative AI-powered teachable agent for middle school mathematics learning: A design-based research study

This paper reports on a design-based research (DBR) study that aims to devise an artificial intelligence (AI)-powered teachable agent that supports secondary school students' learning-by-teaching practices of mathematics learning content. A long-standing pedagogical practice of learning-by-teaching is powered by a recent advancement of generative AI technologies, yielding our teachable agent called ALTER-Math. This study chronicles one usability testing and three cycles of iterative design and implementation process of ALTER-Math. The three empirical studies involved a total of 320 middle school students and six teachers in authentic classroom settings. The first study was exploratory, focusing on the qualitative feedback from the students and teachers through open-ended surveys, interviews and classroom observations. The second study yielded a medium-high (M = 3.26) quantitative survey result on students' perceived engagement and usability on top of the qualitative findings. Finally, the final study included pre- and post-knowledge tests in a quasi-experimental study design as well as student and teacher interviews. The final study revealed a bigger significant knowledge improvement in students who used ALTER-Math compared to the control group, suggesting a positive impact of AI-powered teachable agents on students' learning. The design implications learned from multiple iterations are discussed to inform the future design of AI-powered learning technologies.

Practitioner notes

What is already known about this topic

• Learning-by-teaching is a long-standing effective pedagogical strategy to enhance students' domain knowledge and feelings of responsibility in learning.
• Various teachable agents have been developed and have demonstrated benefits in students' learning.
• Generative AI offers the potential to provide naturalistic, contextualised and adaptive conversations.

What this paper adds

• Develops a novel generative AI-powered teachable agent for middle school mathematics learning, called ALTER-Math.
• Reports the iterative design process involving empirical classroom implementations of ALTER-Math.
• Reveals a bigger significant improvement in the student's mathematical knowledge after using ALTER-Math, compared to the control group.

Implications for practice and/or policy

• Researchers can be inspired by this design example of a theoretically grounded generative AI learning technology.
• Educational technology designers could hear the real voices of students and teachers about the generative AI learning technologies.
• Researchers and educational technology designers could be directed by the design implications to the future design of AI-powered learning technologies and teachable agents.