Modality fusion using auxiliary tasks for dementia detection

Abstract

Alzheimer’s disease is the leading cause of dementia that affects elderly individual’s speech and language abilities. In this paper, a Feature Fusion Model with Guide Patterns (FFG) is designed as an acoustic- and linguistic-based dementia detection system, considering the limited publicly available data and modalities fusion inefficiency. Specifically, a multi-modal features interaction module composed of multiple co-attention layers is designed to improve multi-modal interaction between the acoustic and linguistic information embedded in the audio recordings. Given the limited audio recordings available in public datasets, guide patterns are introduced as auxiliary tasks to enhance the interaction between acoustic and linguistic information. Our proposed FFG model is evaluated with three publicly available datasets, namely, Pitt, ADReSS, and ADReSSo. Experimental results demonstrate that the FFG model can achieve superior resu lts on all three publicly available datasets. An exceptional performance of 85.85% and 84.30% accuracy was achieved on the Pitt and ADReSSo datasets. The ablation study demonstrated the efficiency of our proposed model.