Automated analysis of a novel object recognition test in mice using image processing and machine learning

The novel object recognition test (NORT) is one of the most commonly employed behavioral tests in experimental animals designed to evaluate an animal's interest in and recognition of novelty. However, manual procedures, which rely on researchers’ observations, prevent high throughput analysis. In this study, we developed an automated analysis method for NORT utilizing machine learning-assisted exploratory behavior detection. We recorded the exploratory behavior of the mice using a video camera. The coordinates of the mouse nose and tail base in recorded video files were detected using a pre-trained machine learning model, DeepLabCut. Each video was then segmented into frame images, which were categorized into "exploratory,” or "non-exploratory" frames based on manual observation. Mouse feature vectors were calculated as vectors from the nose to the vertices of the object and were utilized for SVM training. The trained SVM effectively detected exploratory behaviors, showing a strong correlation with human observer assessments. Upon application to NORT, the duration of mouse exploratory behavior towards objects predicted by the SVM exhibited a significant correlation with the assessments made by human observers. The novelty discrimination index derived from the SVM predictions also aligned well with that from human observations.