Evaluating Random Forest Model Performance for Cave and Sinkhole Prediction in the Cradle of Humankind, South Africa: Preliminary Analysis and Variable Importance Assessments

Abstract

Surveying an area for new fossil sites is a labor-intensive and resource-draining activity that can be alleviated with the aid of machine learning models. In karst landscapes of southern Africa, Plio-Pleistocene fossils that inform the paleoanthropological record are primarily found preserved in caves and sinkholes. The purpose of this study is to assess the utility of Random Forest (RF) models for cave and sinkhole prediction in the Cradle of Humankind, South Africa. Multispectral satellite imagery, digital elevation models (DEMs), and geologic maps were converted into raster (pixelated matrix) images in a GIS environment to denote varying aspects of the local topography, including elevation, slope, aspect, curvature, drainage, spectral reflectance, vegetation cover, fault proximity, and underlying geology. The rasters were stacked and overlaid with 1080 known cave and sinkhole locality points and 1080 random non-cave points in the study area for model training. Variable values associated with these geopoints were input into an RF model in Python for training and evaluation using a spatial ten-fold cross-validation. The model performed with 81.6% accuracy and an area under the curve (AUC) of 0.912. The importance of each variable for prediction was evaluated by measuring the increase in prediction error when variable values were shuffled. Distance to major faults, location within the Chuniespoort geologic group, dolomite presence, chert presence, and elevation exhibited the highest importance for model accuracy, while three out of 48 total predictor variables exhibited less importance than a randomly generated variable. The identification of important/unimportant variables will help build more efficient, robust models in future iterations, as well as help identify variables that could be useful in other karst regions.