Exploration of Key Point Localization Neural Network Architectures for Y-Maze Behavior Test Automation

The Y-maze behavioral test is a pivotal tool for assessing the memory and exploratory tendencies of mice in novel environments. A significant aspect of this test involves the continuous tracking and pinpointing of the mouse’s location, a task that can be labor-intensive for human researchers. This study introduced an automated solution to this challenge through camera-based image processing. We argued that key point localization techniques are more effective than object detection methods, given that only a single mouse is involved in the test. Through an experimental comparison of eight distinct neural network architectures, we identified the most effective structures for localizing key points such as the mouse’s nose, body center, and tail base. Our models were designed to predict not only the mouse key points but also the reference points of the Y-maze device, aiming to streamline the analysis process and minimize human intervention. The approach involves the generation of a heatmap using a deep learning neural network structure, followed by the extraction of the key points’ central location from the heatmap using a soft argmax function. The findings of this study provide a practical guide for experimenters in the selection and application of neural network architectures for Y-maze behavioral testing.