Serotonin drives aggression and social behaviors of laboratory male mice in a semi-natural environment

Aggression is an adaptive social behavior crucial for the stability and prosperity of social groups. When uncontrolled, aggression leads to pathological violence that disrupts group structure and individual wellbeing. The comorbidity of uncontrolled aggression across different psychopathologies makes it a potential endophenotype of mental disorders with the same neurobiological substrates. Serotonin plays a critical role in regulating impulsive and aggressive behaviors. Mice lacking in brain serotonin, due to the ablation of tryptophan hydroxylase 2 (TPH2), the rate-limiting enzyme in serotonin synthesis, could serve as a potential model for studying pathological aggression. Home cage monitoring allows for the continuous observation and quantification of social and non-social behaviors in group-housed, freely-moving mice. Using an ethological approach, we investigated the impact of central serotonin ablation on the everyday expression of social and non-social behaviors and their correlations in undisturbed, group-living Tph2-deficient and wildtype mice. By training a machine learning algorithm on behavioral time series, “allogrooming”, “struggling at feeder”, and “eating” emerged as key behaviors dissociating one genotype from the other. Although Tph2-deficient mice exhibited characteristics of pathological aggression and reduced communication compared to wildtype animals, they still demonstrated affiliative huddle behaviors to normal levels. Altogether, such a distinct and dynamic phenotype of Tph2-deficient mice influenced the group's structure and the subsequent development of its hierarchical organization. These aspects were analyzed using social network analysis and the Glicko rating methods. This study demonstrates the importance of the ethological approach for understanding the global impact of pathological aggression on various aspects of life, both at the individual and group levels. Home cage monitoring allows the observation of the natural behaviors of mice in a semi-natural habitat, providing an accurate representation of real-world phenomena and pathological mechanisms. The results of this study provide insights into the neurobiological substrate of pathological aggression and its potential role in complex brain disorders.