A Multimodal Approach for Early Identification of Mild Cognitive Impairment and Alzheimer’s Disease With Fusion Network Using Eye Movements and Speech

Detecting Alzheimer’s disease (AD) in its earliest stages, particularly during an onset of Mild Cognitive Impairment (MCI), remains challenging due to the overlap of initial symptoms with normal aging processes. Given that no cure exists and current medications only slow the disease’s progression, early identification of at-risk individuals is crucial. The combination of eye-tracking and speech analysis offers a promising diagnostic solution by providing a non-invasive method to examine differences between healthy controls and individuals with MCI, who may progress to develop AD. In this study, we analyzed a multimodal clinical eye-tracking and speech dataset collected from 78 participants (37 controls, 20 MCI, and 21 AD) during the King-Devick test and a reading task to classify and diagnose MCI/AD versus healthy controls. To that end, we developed a Fusion Neural Network, a deep learning-based classification model that integrates gaze and speech-derived features, including pupil size variations, fixation duration, saccadic movements, and speech delay, to improve MCI diagnosis performance. We achieved an average classification accuracy of 79.2% for MCI diagnosis and 82% for AD. Our findings indicate that features related to pupil size and eye-speech temporal dynamics are strong indicators for detection tasks. Moreover, the results indicate that using multimodal data (gaze + speech) significantly improves classification accuracy compared to unimodal data from speech or gaze alone.