YOLO-MN: High-Throughput cell micronucleus detection based on prior knowledge and frequency domain perception

Cell micronuclei, key indicators of chromosomal damage, have broad applications in environmental toxicology, radiation research, and drug safety testing. Existing deep learning based micronuclei detection face numerous challenges including morphological similarity between micronuclei and cell nuclei, large size differences, and loss of detail information during multi-scale feature fusion. To address these challenges, this paper proposes an innovative cell micronucleus detection method termed YOLO-MN based on prior knowledge and frequency domain perception. We designed a wavelet feature enhancement (WFE) module based on two-dimensional discrete wavelet transform, which extracts multi-scale features through wavelet decomposition and enhances edge and chromatin texture information of micronuclei during feature fusion. Next, we designed a micronucleus attention module (MAM) and improved CSP feature extraction network (C2f_MAM) based on biological prior knowledge of micronuclei, focusing on shape features and capturing spatial relationships between micronuclei and the main nucleus. Finally, we designed an MNIoU loss function incorporating micronucleus prior knowledge to accelerate model convergence and further improve detection accuracy. Experimental results show that YOLO-MN achieved 91.7% Precision, 93.4% Recall, 94.7% mAP@50, and 59.1% mAP@50–95 on the cell micronucleus dataset, the model’s generalization capability was further validated on the SRCHD and LISC datasets.