Automating the Assessment of Problem-solving Practices Using Log Data and Data Mining Techniques

Abstract

Interactive simulations provide an exciting opportunity to assess and teach students the practices used by scientists and engineers to solve real-world problems. This study examines how the logged interaction data from a simulation-based task could be used to automate the assessment of complex problem-solving practices. A total of 73 college students worked on an interactive circuit puzzle embedded in a science simulation in an interview setting. Their problem-solving processes were videotaped and logged in the backend of the simulation. We extracted different sets of features from the log data and evaluated their effectiveness as predictors of students' problem-solving success and evidence for specific problem-solving practices. Our results indicate that the application of data mining techniques guided by knowledge gained from qualitative observation was instrumental in the discovery of semantically meaningful features from the raw log data. These knowledge-grounded features were significant predictors of students' overall problem-solving success and provided evidence on how well they adopted specific problem-solving practices, including decomposition, data collection, and data recording. The results point to promising directions for how scaffolding/feedback could be provided in educational simulations to enhance student learning in problem-solving skills.