Ai-assissted multimodal classification of canine and feline (Sub-)Cutaneous tumors using ultrasound, white light and fluorescence imaging

Abstract

Research aims to provide an automated method for the diagnostics of (sub-)cutaneous tumors in canine/feline patients via machine learning (ML) on multimodal imaging data. To address the limitations of single mode imaging, we have integrated optical imaging, white light (WL) and fluorescence (FL), in tandem with high frequency (∼350 MHz) ultrasound (US) imaging. Combined hybrid approaches allowed to efficiently differentiate between malignant mastocytomas (MCTs), sarcomas (STSs) and benign lipomas (LPs).

The obtained combinations of spectral and temporal US estimates were evaluated for multimodal effect when grouped with FL, WL and optics (FL + WL), resulting in hybrid bimodal and trimodal datasets. Support vector machine (SVM) and random forest (RF) classifiers were optimized via sequential feature selection, resulting in multimodal effect (p < 0.05).

Pronounced multimodal increment was detected for US and FL (up to 15 %) for 21–23/31 US combinations for total of 360 MCT and STS cases. US + WL and US + OPT resulted in less efficient multimodal effect (up to ∼5 %). Both SVM and RF classifiers strongly preferred US features in combination with FL. In the presence of superior WL, the relevance of US features decreased, however, still, they comprised more than a quarter of the total number of features.

In overall, we have achieved classification efficiency in the range of: 75–80 % (US + FL), 85–90 % (US + WL) as well as 90–95 % (US + OPT), respectively. Our approach provides a reliable diagnostic support for veterinary histopathologist in real-time; therefore, if a disagreement between human and ML-derived analysis occurrs, unclear cases can be double-inspected by human.