Enhancing collaboration between teacher and student for effective cross-domain nuclei detection and classification

Automated detection and classification of cell nuclei in histopathology images is critical for accurate cancer diagnosis. Deep learning-based methods have shown promise, yet their effectiveness is often undermined by domain shift arising from variations in patient data, staining protocols, and imaging devices between training and testing datasets. The teacher-student framework has emerged as a viable strategy for domain adaptation, wherein the teacher transfers source domain knowledge to the student. However, the framework is vulnerable to unreliable pseudo label, which can further lead to a vicious cycle of propagating incorrect information between teacher and student. In this study, we present the Collaborative Teacher-Student (CTS) framework for cross-domain nuclei detection and classification, which is intended to assist in diagnosing various types of cancer. The CTS introduces the Identity Swap Mechanism (ISM) that dynamically exchanges the identities of teacher and student models based on their respective performance. This mechanism fosters a mutual learning paradigm, effectively mitigating the propagation of misinformation and preventing performance degradation. Additionally, we propose the Joint Uncertainty-guided Student Training (JUST) strategy that incorporates uncertainty estimates from both teacher and student models, to filter out unreliable pseudo labels and facilitate more accurate knowledge transfer. Experimental results demonstrate that the CTS framework consistently outperforms existing methods across multiple domain adaptation scenarios. Notably, it achieves significant performance improvements of 3.1 % in detection F-score and 2.4 % in classification F-score on the breast cancer dataset BCNuP. The code will be made available at: https://github.com/waq2001/collaborative_teacher.