Common to rare transfer learning (CORAL) enables inference and prediction for a quarter million rare Malagasy arthropods

Abstract

DNA-based biodiversity surveys result in massive-scale data, including up to millions of species—of which, most are rare. Making the most of such data for inference and prediction requires modeling approaches that can relate species occurrences to environmental and spatial predictors, while incorporating information about their taxonomic or phylogenetic placement. Even if the scalability of joint species distribution models to large communities has greatly advanced, incorporating hundreds of thousands of species has not been feasible to date, leading to compromised analyses. Here we present a ‘common to rare transfer learning’ (CORAL) approach, based on borrowing information from the common species to enable statistically and computationally efficient modeling of both common and rare species. We illustrate that CORAL leads to much improved prediction and inference in the context of DNA metabarcoding data from Madagascar, comprising 255,188 arthropod species detected in 2,874 samples. CORAL can infer the occurrence of rare species based on common species, using DNA metabarcoding data or other high-dimensional biodiversity data. The approach is illustrated on a large-scale biodiversity survey from Madagascar.