Automated phenotyping of rodent behavior in the Cylinder Exploration Test using machine learning

Abstract

Rodent models are essential in neuroscience research for investigating brain function, CNS disease mechanisms, and therapeutic interventions. Beyond molecular and physiological analyses, precise behavioral characterization provides crucial functional readouts of neural circuit changes. Accurate behavioral phenotyping is critical for detecting genotype-phenotype relationships, enabling cross-model comparisons, and supporting translational research. Here we developed a machine learning framework for automated rodent behavior analysis in the Cylinder Exploration Test (CET) using pose estimation and explainable machine learning. The framework quantifies freezing, rearing, exploratory movement, and general locomotion activity while identifying key behavioral features that differentiate between experimental conditions. To validate this approach, we phenotyped two rat strains with dopaminergic and serotonergic dysfunction: dopamine transporter knockout (DAT-KO), tryptophan hydroxylase 2 knockout (Tph2-KO), and their wild-type controls. The analysis successfully identified distinct strain-specific behavioral phenotypes and characterized the discriminative features between genotypes, achieving high classification accuracy (AUC = 0.84 for DAT-KO versus DAT-WT and AUC = 0.98 for Tph2-KO versus Tph2-WT). These findings demonstrate that automated Cylinder Exploration Test can detect genotype-specific signatures and establish a scalable method for standardized phenotyping in neuroscience and preclinical research.