Diagnose Like a Doctor: A Vision-Guided Global–Local Fusion Network for Chest Disease Diagnosis

Abstract

Chest diseases are the most common diseases around the world. Deep neural networks for chest disease diagnosis are usually limited by the need for extensive manual labeling and insufficient model interpretability. To this end, we propose the dual-branch framework called Vision-Guided global–local fusion network (VGFNet) for chest disease diagnosis like an experienced doctor. We first introduce radiologists' eye-tracking data as a low-cost but easily accessible information source, which implicitly contains sufficient but unexplored pathological knowledge that provides the localization of lesions. An eye-tracking network (ETNet) is first devised to learn clinical observation patterns from the eye-tracking data. Then, we propose a dual-branch network that can simultaneously process global and local features. ETNet provides the approximate local lesions to guide the learning procedure of the local branch. Meanwhile, a triple convolutional attention (TCA) module is created and embedded into the global branch to refine the global features. Finally, a convolution attention fusion (CAF) module is designed to fuse the heterogeneous features from the two branches, taking full advantage of their local and global representation abilities. Extensive experiments demonstrate that VGFNet can significantly improve classification performance on both multilabel classification and multiclassification tasks, obtaining an AUC value of 0.841 on Chest x-ray14 and an accuracy of 0.9820 on RAD, which outperforms state-of-the-art models. We also validate the model's generalizability on Chest x-ray. This study introduces eye-tracking data, which increases the interpretability of the model and provides new perspectives for deep mining of eye-tracking data. Meanwhile, we designed several plug-and-play modules to provide new ideas in the field of feature refinement. The code for our model is available at https://github.com/ZXJ-YeYe/VGFNet.