Multimodal learning analytics for students behavior prediction using multi-scale dilated deep temporal convolution network with improved chameleon Swarm algorithm

Despite the necessary knowledge, a variety of aspects affect the performance of the students on a task. Students’ on-task efforts have been noted to be significantly connected with their academic performance, demonstrating how effectively the students are involved in that task. Yet, these efforts given by students to perform a task cannot be seen directly. Multimodal knowledge may enable assessment of the student’s effort and offer extra perspectives into active learning. To provide new perspectives into the processes of learning premised on obtaining goals like behavioral trajectories, students’ achievement, support from teachers, feedback from students, involvement, and learning-task efficiency, a new deep learning-based multi-modal framework for data analytics of students’ behaviors are developed. At first, the multi-modal information is gathered from the relevant standard data sources, including video, sound, texts, EEG, eye movements, and facial information. For extracting the deep characteristics from the multi-modal input, the One Dimensional Convolutional Neural Network (1D-CNN) is utilized for feature extraction from audio, the Three Dimensional CNN (3D-CNN) is utilized for feature extraction from video, and Transformer-net is used for feature extraction from text. The proposed Modified Random Parameter-based Chameleon Swarm Algorithm (MRP-CSA) is used to optimize the weights before performing the weighted feature selection. The chosen characteristics are fed into the Adaptive Multi-scale Dilated Deep Temporal Convolution Network (AMDDTCN), which is utilized to identify the behavior of students while considering the expectations of the students, which also influences their involvement in the overall learning behavior evaluation stage. The implemented MRP-CSA is used to optimize the parameters inside the AMDDTCN during this evaluation phase. To ensure that the generated model is effective, the trial results are compared to the current multi-modal data analytics model. The developed model is evaluated using various performance metrics such as accuracy, precision, specificity, F1-score, precision, False Positive rate(FPR), False Negative Rate(FNR), Negative Predictive Value (NPV), False Discovery Rate (FDR) and Matthews Correlation Coefficient (MCC), and given the accuracy to be 97.2%. Thus, it is proved that the proposed model is promising over other traditional methods and has more abilities in identifying students’ behaviors by using a deep learning model with an optimization algorithm.