FeatureForest: the power of foundation models, the usability of random forests.

Abstract

Analysis of biological images relies heavily on segmenting the biological objects of interest in the image before performing quantitative analysis. Deep learning (DL) is ubiquitous in such segmentation tasks, but can be cumbersome to apply, as it often requires a large amount of manual labeling to produce ground-truth data, and expert knowledge to train the models. More recently, large foundation models, such as SAM, have shown promising results on scientific images. They, however, require manual prompting for each object or tedious post-processing to selectively segment these objects. Here, we present FeatureForest, a method that leverages the feature embeddings of large foundation models to train a random forest classifier, thereby providing users with a rapid way of semantically segmenting complex images using only a few labeling strokes. We demonstrate the improvement in performance over a variety of datasets and provide an open-source implementation in napari that can be extended to new models.