Safe AI for coral reefs: Benchmarking out-of-distribution detection algorithms for coral reef image surveys

Abstract

Although deep learning has demonstrated significant advances in qualitative domains, deep learning algorithms remain poor at quantifying the uncertainty of their predictions. This is especially true when applied in domains where there is data shift outside of which an algorithm has been trained. This has major implications for the use of deep learning in accurately estimating change in environmental monitoring applications. In the case of image classification for coral reef habitats, time series imagery is rarely consistent due to changing environmental conditions, differing sensors and inconsistencies in data capture, which means traditional machine learning metrics simply do not work when applied to new out of distribution datasets.

• 1.
  For this reason, we benchmark the latest state-of-the-art OOD (Out Of Distribution) detection algorithms on publicly available coral reef image datasets, and evaluate histogram intersection of outlier scores as an indicator for human intervention.
• 2.
  We show with a comparative analysis that the performance of OOD detection algorithms is variable, and highly dependent on in-distribution and out-of-distribution data composition. We show that KNN (K-Nearest Neighbour) distance was the most consistent across datasets, followed by Virtual-logit matching (ViM).
• 3.
  This research shows a compelling example of how a handful of openly available algorithms can easily be used as a complimentary indicator alongside confidence (Softmax probability), in turn providing more efficient and safe deployment of deep learning for rapid reporting of coral reef habitats.