Evaluating the performance of an artificial intelligence-based electronic reader for malaria rapid diagnostic tests across Benin, Côte d'Ivoire, Nigeria and Uganda.

Abstract

Background: The introduction of malaria rapid diagnostic tests (RDTs) has expanded the parasitological confirmation of malaria at all levels of health systems in sub-Saharan Africa, improving case management and surveillance. However, concerns persist regarding healthcare worker adherence to RDT outcomes and the accuracy of RDT results recorded in health facility registers. Electronic RDT readers have been proposed to improve the consistency of interpretation and reporting. The HealthPulse smartphone application (Audere, Seattle, WA, USA), an RDT reader using an artificial intelligence (AI) computer vision algorithm, was assessed against a trained human panel interpreting RDT results from photographs to determine the application's performance characteristics.

Methods: In 2023, the Malaria Rapid Diagnostic Test Capture and Reporting Assessment (MaCRA) was implemented in health facilities in Benin, Côte d'Ivoire, Nigeria, and Uganda. Study staff photographed malaria RDTs using the HealthPulse application after healthcare workers performed and interpreted the tests. A trained panel of external reviewers interpreted the RDT images and served as the reference standard. RDTs in the images were classified according to the manufacturer's instructions as positive, negative or invalid (i.e., no visible control line) or labelled as uninterpretable (i.e., visibility was impeded). The performance of the HealthPulse AI algorithm was evaluated using percent accuracy, recall (i.e., sensitivity and specificity), precision (i.e., positive and negative predictive values), and F1 scores (harmonic mean of recall and precision) weighted by the number of each outcome. Logistic regression was applied to assess factors influencing recall across countries, RDT products, presence of faint lines, and anomalies.

Results: Of the 110,843 RDT images collected, 106,877 (96.4%) were included in the analysis. The AI algorithm demonstrated high accuracy (96.8%; 95% confidence interval (CI) 96.7%, 96.9%) compared with the panel interpretation and an overall F1 score of 96.6. Recall and precision were > 97% for positive and negative outcomes but much lower for invalid (recall: 84.8%; precision: 42.8%) and uninterpretable (recall: 0.8%; precision: 2.3%) classifications. AI performance varied by country, RDT product, the presence of faint lines and the quality of the image. When test lines were faint, the AI algorithm was significantly less likely to recall both positive results (adjusted odds ratio (aOR) 0.02; 95% CI 0.02, 0.02) and negative results (aOR 0.10; 95% CI 0.07, 0.16).

Conclusions: The HealthPulse AI algorithm demonstrated strong agreement with a trained panel in interpreting malaria RDT images across diverse settings. However, the reduced performance for invalid outcomes and varying performance by country, RDT product and faint lines highlight the need for further research and refinement. The HealthPulse application shows potential as a supportive tool in research, training, surveillance, and quality assurance.