Comparing Machine Learning and Human Judge in SATU Indonesia Awarding Processes

Abstract

Received November 09, 2021 Revised November 25, 2021 Accepted December 23, 2021 For more than ten years, SATU Indonesia Awards, with PT. Astra International Tbk's support is given to inspiring young Indonesians. Every year, more than 10,000 nominations must be short-listed to 90 nominations within one week with five (5) assessment parameters. The research contributions are (1) creating a machine learning mechanism for the awarding process from ten years of the SATU Indonesia Awards nomination archive, (2) creating two (2) models of training data for the five (5) assessed parameters, namely motivation, obstacle, outcome, outreach, and sustainability, and (3) compare machine learning prediction with 2021 judge's assessment. TEMPO Data and Analysis Center (PDAT) extracts the corpus training data from ten years' SATU Indonesia Awards data in six months. The corpus training data contains nomination texts with Judges' scores on motivation, obstacle, outcome, outreach, and sustainability. Two (2) corpus training data and two models were generated with, namely, (1) the average Judges' parameter value per instance and (2) the Judges' smallest value and stored in two (2) corpus of 1220 instances each. The classification model was generated by Random Forest, which has the slightest error among the classification algorithms tested. The first model aims to predict the nomination assessment parameters. The second model is to detect the outlier in the incoming nominees for extraordinary nominees. The machine learning predictions were compared and found to be similar to the 2021 judge's assessment in the awarding processes at SATU Indonesia Awards. The average Judges' pre-final 2021 nominees' scores are compared to the Random Forest's predictions and found to be reasonably similar, with a small RMSE error around 1.1 to 1.6 for all assessment parameters. The smallest RMSE was obtained in the Sustainability parameter. The Obstacle parameter was found to have the largest RMSE.