Biology-driven insights into the power of single-cell foundation models

Single-cell foundation models (scFMs) have emerged as powerful tools for integrating heterogeneous datasets and exploring biological systems. Despite high expectations, their ability to extract unique biological insights beyond standard methods and their advantages over traditional approaches in specific tasks remain unclear. Here, we present a comprehensive benchmark study of six scFMs against well-established baselines under realistic conditions, encompassing two gene-level and four cell-level tasks. Pre-clinical batch integration and cell type annotation are evaluated across five datasets with diverse biological conditions, while clinically relevant tasks, such as cancer cell identification and drug sensitivity prediction, are assessed across seven cancer types and four drugs. Model performance is evaluated using 12 metrics spanning unsupervised, supervised, and knowledge-based approaches, including scGraph-OntoRWR, a novel metric designed to uncover intrinsic knowledge encoded by scFMs. We provide holistic rankings from dataset-specific to general performance to guide model selection. Our findings reveal that scFMs are robust and versatile tools for diverse applications while simpler machine learning models are more adept at efficiently adapting to specific datasets, particularly under resource constraints. Notably, no single scFM consistently outperforms others across all tasks, emphasizing the need for tailored model selection based on factors such as dataset size, task complexity, biological interpretability, and computational resources. This benchmark introduces novel evaluation perspectives, identifying the strengths and limitations of current scFMs, and paves the way for their effective application in biological and clinical research, including cell atlas construction, tumor microenvironment studies, and treatment decision-making.