Utilization of Image-Based Deep Learning in Multimodal Glaucoma Detection Neural Network from a Primary Patient Cohort

Abstract

Purpose

To develop a clinically motivated multimodal neural network glaucoma detection model trained on minimally processed imaging data of time-matched multimodal testing including fundus photographs, OCT scans, and Humphrey visual field (HVF) analysis.

Design

Evaluation of a diagnostic technology.

Subjects

A total of 716 encounters with time-matched fundus photographs, OCT optic nerve imaging, and HVF testing from 706 eyes (557 nonglaucomatous, 149 glaucomatous) from 571 individual patients seen at a tertiary medical center and 4 external single-modality (fundus photograph and OCT) datasets.

Methods

A multimodal neural network model was developed consisting of 2 main components: first, 3 convolutional neural networks to extract semantic features and generate embeddings for each respective modality, followed by a second component consisting of a multilayer perceptron to integrate the individual embeddings and produce a predicted label, glaucomatous or nonglaucomatous.

Main Outcome Measures

Single and multimodal performances were evaluated on the internal test set using the area under the receiver operating characteristic curve (AUC), accuracy, recall, and specificity. Fundus photograph and OCT single-modality neural networks were additionally evaluated on external datasets by these metrics.

Results

Our results show single-modality models with high performance on curated training datasets perform inferiorly on our primary clinical dataset. Performance metrics, however, can be notably improved through multimodal integration (AUC: 0.86 from 0.57 to 0.74 and specificity: 0.85 from 0.77 to 0.82), suggesting that a holistic approach considering both structural and functional data may enhance the functionality and accuracy of artificial intelligence (AI) model.

Conclusions

Clinical implementation of deep learning models for glaucoma detection benefits from multimodal integration, and we demonstrate this approach on a true clinical cohort to obtain a production-level AI solution for glaucoma diagnosis.

Financial Disclosure(s)

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.