Reference aware attention based medical image diagnosis

Abstract

Given the excellent globality and parallelism, Transformer has been widely applied to image tasks. Visual Transformers demand modeling the spatial correlations among visual tokens. However, those existing methods either only emphasize the relative position between two tokens, or only concern on their contexts. Intuitively, a rational attention distribution should hinge on both. To this end, this paper proposes Reference Aware Attention (RAA). RAA decomposes inner-tokens dependency into three intuitive factors, in which reference bias is introduced to model how a reference token attends to a region. Experimental results suggest that RAA can effectively promote the performances of visual Transformers on various medical image diagnosis tasks.