On bootstrapping, data overfitting and crocodiles: an additional comment to McPherron et al. (2022)

Abstract

Quaternary hominin-carnivore interactions is taphonomically reconstructed best through the use of bone surface modifications (BSM). This study examines redundancy in an experimental dataset of potentially similar BSM created by crocodile tooth-marking, sedimentary trampling and stone tool cut marking (Domínguez-Rodrigo and Baquedano in Sci Rep 8:5786, 2018). The original analysis of this experimental set, aiming to confidently classify the three types of BSM, was criticized by some authors (McPherron et al. in J Hum Evol 164:103071, 2022) insinuating that the analysis was flawed by a potential methodological overfitting caused by the improper use of bootstrap. A subsequent response to that critique (Abellán et al. in Geobios Memoire Special. 72–73, 12–21, 2022) showed that there was no difference in the results between using the raw data and the bootstrapped data. It was argued that structural co-variance and redundancy of the categorical dataset was responsible for the highly accurate models; however, this was never empirically demonstrated. Here, we show how the original experimental dataset is saturated with redundancy. Our analysis revealed that, out of 633 cases, only 116 were unique (18.3%) in the complete dataset, 45 unique cases (7.1%) in the intrinsic variable dataset, and just four unique cases (0.63%) in the three-variable dataset (accounting for most of the sample variance). Redundancy, therefore, ranged from 81.7% to over 99%. Machine learning analysis using Random Forest (RF) and C5.0 algorithms on the datasets demonstrated high accuracy with the raw data (90-98%). Proper bootstrapping yielded nearly identical accuracy (88-98%), while improper bootstrapping slightly reduced accuracy (86-98%) and introduced some degree of underfitting. This underscores that the potential biasing effects of bootstrapping differ between numerical and categorical datasets, especially on those with low dimensionality and low cardinality, in situations of feature interdependence and covariance. A complementary approach, consisting of an iterative data partitioning method through train-test resampling reproduced the results derived from the bootstrapped samples. The understanding of these methodological processes is essential to an adequate application of these experimental models to the fossil record.