Enhancing the Admissions Process for Engineering Baccalaureate Programs: A Machine Learning Approach to Developing a Valid and Reliable Examination

This paper details an approach to identify multiple-choice questions that are most effective in discriminating deficiencies in mathematics competencies of incoming first-year engineering students who are graduates of the K-12 program that was recently implemented in the Philippines. To achieve this objective, machine learning algorithms such as the k-Nearest Neighbors (kNN), Logistic Regression (LR), Random Forest (RF), Decision Tree (DT), and Gradient Boosting Machines (GBM) were implemented. From a question bank containing 1,300 questions covering Algebra (A), Advanced Algebra (AA), Plane and Spherical Trigonometry (T), Analytic Geometry (AG), and Solid Mensuration (SM), five domain experts identified the suitability of the questions as part of a diagnostic examination in mathematics. Specifically, using a 5-point Likert scale (5 being the highest), the experts rated the suitability of each question to test the proficiency of a student in 23 mathematics competencies based on what is prescribed by the Commission on Higher Education (CHED). The collected survey data were then used to train the machine learning models, which extracted patterns to identify the questions that would be most suitable to test the mathematics competencies of incoming first-year engineering students. With a precision recall score of 99.90%, the LR model was selected as the best performing model and analysis of how the LR model predicts the labels through the use of shap values revealed that the preference was given towards questions which test student proficiency in foundational mathematics competencies like that of Algebra and Analytical Geometry. Overall, these findings provided a better understanding of the questions that are most effective in discriminating student deficiencies in mathematics subjects.