Studying the relationship between BKT fitting error and the skill difficulty index

Bayesian Knowledge Tracing (BKT) is one of the most popular knowledge inference models due to its interpretability and ability to infer student knowledge. A proper student modeling can help guide the behavior of a cognitive tutor system and provide insight to researchers on understanding how students learn. Using four different datasets we study the relationship between the error coming from fitting the parameters and the difficulty index of the skills and the effect of the size of the dataset in this relationship. The relationship between the fitting error and the difficulty index can be very easy modeled and might be indicating some problems with BKTs performance. However, large datasets are required to clearly see this connection as there is an important sample size effect.