Hashimoto’s thyroiditis recognition from multi-modal data via global cross-attention and distance-aware training

Ultrasound images and biological indicators, which reveal Hashimoto’s thyroiditis (HT) characteristics in thyroid tissue from different perspectives, play crucial roles in HT recognition. Ultrasound images of patients with HT typically present a heterogeneous background with potential decreases in echogenicity. Clinicians are prone to misdiagnosing HT by visually evaluating these characteristics. In addition, patients with HT may exhibit fluctuations in relevant biological indicators, but there are no absolute relationships between a single biological indicator and HT. To address these challenges, we propose HTR-Net, a novel HT recognition network that combines ultrasound images and biological indicators through multi-modality information embedding. Specifically, HTR-Net introduces a global cross-attention module (GCA), which enhances recognition of the heterogeneous background with potential decreases in echogenicity. A distance-aware mismatched augmentation (DMA) strategy is also designed to expand the limited biological indicator data and ensure reasonable values for the augmented biological indicators, thus enhancing the model performance. In order to address the nonabsolute relationship between HT and a single biological indicator, we propose a distance-aware loss (DL) function to constrain feature mapping for effective information extraction from indicators, thereby enhancing the model’s capability to detect anomalous sets of biological indicators. To validate the proposed method, we construct a multi-center HT dataset and conduct extensive experiments. The experimental results demonstrate that the proposed HTR-Net achieves state-of-the-art (SOTA) performance.