Automated Data Driven Preprocessing and Training of Classification Models

Our work is a distributed machine learning pipeline designed to scale to large datasets. It aims to automate the entire process of solving a classification problem. It just requires the dataset and the target column as an input, and then the system takes care of the rest. Efficient cleaning of the dataset is performed, which imputes all the missing values and gives better structure to the dataset. The system is capable of detecting categorical values, thus performing One-Hot Encoding where required. Further, in the preprocessing stage, it also takes care of feature engineering, dimensionality reduction, sampling, and removal of outliers which affect the model's accuracy. After the preprocessing phase, the ready data is trained on several models, with multiple different hyperparameters. The system's output is the name, accuracy, and code of the best model, which is judged based on its accuracy. The system is tested on over 30 datasets, both binary and multi-class classification, and there is a robust system to train any dataset given to it quickly.