Are we there yet? Reliable occupancy modeling from AI-labeled trail camera data

Abstract

Computer vision is increasingly adopted to automate wildlife image classification, but its ecological reliability remains largely untested. We evaluated SpeciesNet, a global wildlife classifier on widespread mammal species of Florida, USA: the white-tailed deer (Odocoileus virginianus), wild pig (Sus scrofa), coyote (Canis latrans) and bobcat (Lynx rufus). While conventional computer-vision evaluation scores were high (e.g., F1 = 98.2% for the white-tailed deer), they masked species-specific false-positive rates (FPRs) that drove bias in downstream occupancy estimates. The white-tailed deer with FPR = 0.14% produced the largest divergence from the human-labeled data, whereas bobcat, with a lower F1 (94.5%) but FPR = 0.01%, yielded near-identical estimates. We show that FPRs, not conventional metrics like F1-scores, are key factors for occupancy estimate accuracy when using standard models. Our findings highlight the limitations of conventional performance metrics when evaluating AI tools for ecological applications and the need of false-positive-adequate methods as AI-classifiers gain popularity. Our analysis shows that false-positive-informed occupancy models quickly outperform standard models as FPRs increase. Standard models still retain an advantage at low false positive rates (<0.35% in our simulation conditions) most notably when detection probability is low. The benefits of false-positive occupancy models are highest for species with low occupancy (ψ = 0.3) where false positives are more likely to randomly occur in unoccupied sites. Given that currently available species classifiers likely yield FPRs higher than 0.34%, our analysis indicates that false-positive-informed occupancy models would outperform standard occupancy models in analyses using partially human-verified AI-labeled data.