Investigating perceived fairness of AI prediction system for math learning: A mixed-methods study with college students

Abstract

Entities such as governments and universities have begun using AI for algorithmic decision-making that impacts people's lives. Despite their known benefits, such as efficiency, the public has raised concerns about the fairness of AI's decision-making. Here, the concept of perceived fairness, defined as people's emotional, cognitive, and behavioral responses toward the justice of the AI system, has been widely discussed as one of the important factors in determining technology acceptance. In the field of AI in education, students are among the biggest stakeholders; thus, it is important to consider students' perceived fairness of AI decision-making systems to gauge technology acceptance. This study adopted an explanatory sequential mixed-method research design involving 428 college students to investigate the factors that impact students' perceived fairness of AI's pass-or-fail prediction decisions in the context of math learning and suggest ways to improve the perceived fairness based on students' voices. The findings suggest that students who received a favorable prediction outcome (i.e., pass), who were presented with a system that had a lower algorithmic bias and higher transparency, who major(ed) in STEM (vs. non-STEM), who have higher math anxiety, and who received the outcome that matches their math knowledge level (i.e., accurate) tend to report a higher level of perceived fairness for the AI's prediction decisions. Interesting interaction effects were also found regarding decision-making, students' math anxiety and knowledge, and the outcome's favorability on students' perceived fairness. Qualitative thematic analysis revealed students' strong desire for transparency with guidance, explainability, and interactive communication with the AI system, as well as constructive feedback and emotional support. This study contributes to the development of a justice theory in the era of AI and suggests practical design implications for AI systems and communication strategies with AI systems in education.