Automated multimodal severity assessment of diabetic retinopathy using ultra-widefield color fundus photography and clinical tabular data

This study introduces an automatic deep-learning-based approach to diabetic retinopathy (DR) severity assessment by integrating two modalities: Ultra-Widefield Color Fundus Photography (UWF-CFP) from the CLARUS 500 device (Carl Zeiss Meditec Inc., Dublin, CA, USA) and a comprehensive set of clinical data from the EVIRED project. We propose a framework that combines the information from 2D UWF-CFP images and a set of 76 tabular features, including demographic, biochemical, and clinical parameters, to enhance the classification accuracy of DR stages. Our model uses advanced machine learning techniques to address the complexities of synthesizing heterogeneous data types, providing a holistic view of patient health status. Results indicate that this fusion outperforms traditional methods that rely solely on imaging or clinical data, suggesting a robust model which can provide practitioners with a supportive second opinion on DR severity, particularly useful in screening workflows. We measured a multiclass accuracy of 63.4% and kappa of 0.807 for our fusion model which is 2.1% higher in accuracy and 0.022 higher in kappa compared to the image unimodal classifier. Several interpretation methods are used to provide practitioners with an inside view of the workings of classification methods and allow them to discover the most important clinical features.