Evaluating machine learning methods on a large-scale of in silico fire debris data

Abstract

A large dataset of 240,000 fire debris samples have been generated in-silico using a data augmentation method at National Center for Forensic Science. The IS samples contain balanced data with 50 % samples having ignitable liquid residue and 50 % only having substrate components. In the big data era, this large dataset is useful for researchers to develop and implement their new machine learning methods. In this paper, we split the data into a training dataset and a test dataset. We then trained seven machine learning methods including logistic regression, least discriminant analysis, quadratic discriminant analysis, support vector machine, random forest, XGBoost, and neural network on an in-silico training dataset. The predictive accuracy and area under the ROC (AUC) of the models was evaluated and compared on both an in-silico test dataset and on an experimental fire debris dataset. In addition, we analyzed both TIS and TIC datasets. For the TIS dataset, neural network provides the highest AUC in both in-silico test and experimental fire debris dataset. Random forest shows the highest performance for the TIC dataset when we binned the retention index.