Machine Learning Models for Automatic Classification of Programming Tasks in Python

Abstract

This article presents a comparative analysis of automatic error classification in Python programming tasks involving loops, with a focus on machine learning approaches for identifying error types. Three experiments were conducted: the first evaluated traditional machine learning algorithms; the second explored alternative kernel functions in SVM and MLP algorithms; and the third involved two multi-input deep learning models. Following the CRISP-DM methodology, we compiled a dataset of 3000 looped programming tasks, each including a problem description in English, a Python solution, and a classification based on state errors formulated by Gries theory (initial S, final E, state transformation T, and their combinations, totaling seven labels). In preprocessing, the code was partitioned using its AST to align it with Gries states, and both the problem descriptions and code snippets were embedded with Bert, Code-bert, and Graph-code-bert. Ablations were performed to approximate the optimal solutions for each algorithm as closely as possible. The results showed that traditional machine learning algorithms achieved up to 90% accuracy and an MCC of 83%, while alternative kernels marginally improved performance but did not surpass that of traditional algorithms. Deep learning models (ANN-GCB) achieved the best balance, with 94% accuracy and 91% MCC, respectively, demonstrating their superiority for this classification task. The model was validated with external sources from different repositories, yielding an 85% success rate, and with a group of 138 students, achieving a 96.7% correct or partially correct classification rate.