Software with artificial intelligence-derived algorithms for detecting and analysing lung nodules in CT scans: systematic review and economic evaluation.

Background: Lung cancer is one of the most common types of cancer and the leading cause of cancer death in the United Kingdom. Artificial intelligence-based software has been developed to reduce the number of missed or misdiagnosed lung nodules on computed tomography images.

Objective:  To assess the accuracy, clinical effectiveness and cost-effectiveness of using  software with artificial intelligence-derived algorithms to assist in the detection and analysis of lung nodules in computed tomography scans of the chest compared with unassisted reading.

Design: Systematic review and de novo cost-effectiveness analysis.

Methods: Searches were undertaken from 2012 to January 2022. Company submissions were accepted until 31 August 2022. Study quality was assessed using the revised tool for the quality assessment of diagnostic accuracy studies (QUADAS-2), the extension to QUADAS-2 for assessing risk of bias in comparative accuracy studies (QUADAS-C) and the COnsensus-based Standards for the selection of health status Measurement INstruments (COSMIN) checklist. Outcomes were synthesised narratively. Two decision trees were used for cost-effectiveness: (1) a simple decision tree for the detection of actionable nodules and (2) a decision tree reflecting the full clinical pathways for people undergoing chest computed tomography scans. Models estimated incremental cost-effectiveness ratios, cost per correct detection of an actionable nodule, and cost per cancer detected and treated. We undertook scenario and sensitivity analyses.

Results: Twenty-seven studies were included. All were rated as being at high risk of bias. Twenty-four of the included studies used retrospective data sets. Seventeen compared readers with and without artificial intelligence software. One reported prospective screening experiences before and after artificial intelligence software implementation. The remaining studies either evaluated stand-alone artificial intelligence or provided only non-comparative evidence. (1) Artificial intelligence assistance generally improved the detection of any nodules compared with unaided reading (three studies; average per-person sensitivity 0.43-0.68 for unaided and 0.79-0.99 for artificial intelligence-assisted reading), with similar or lower specificity (three studies; 0.77-1.00 for unaided and 0.81-0.97 for artificial intelligence-assisted reading). Nodule diameters were similar or significantly larger with semiautomatic measurements than with manual measurements. Intra-reader and inter-reader agreement in nodule size measurement and in risk classification generally improved with artificial intelligence assistance or were comparable to those with unaided reading. However, the effect on measurement accuracy is unclear. (2) Radiologist reading time generally decreased with artificial intelligence assistance in research settings. (3) Artificial intelligence assistance tended to increase allocated risk categories as defined by clinical guidelines. (4) No relevant clinical effectiveness and cost-effectiveness studies were identified. (5) The de novo cost-effectiveness analysis suggested that for symptomatic and incidental populations, artificial intelligence-assisted computed tomography image analysis dominated the unaided radiologist in cost per correct detection of an actionable nodule. However, when relevant costs and quality-adjusted life-years from the full clinical pathway were included, artificial intelligence-assisted computed tomography reading was dominated by the unaided reader. For screening, artificial intelligence-assisted computed tomography image analysis was cost-effective in the base case and all sensitivity and scenario analyses.

Limitations: Due to the heterogeneity, sparseness, low quality and low applicability of the clinical effectiveness evidence and the major challenges in linking test accuracy evidence to clinical and economic outcomes, the findings presented here are highly uncertain and provide indicators/frameworks for future assessment.

Conclusions: Artificial intelligence-assisted analysis of computed tomography scan images may reduce variability of and improve consistency in the measurement and clinical management of lung nodules. Artificial intelligence may increase nodule and cancer detection but may also increase the number of patients undergoing computed tomography surveillance unnecessarily. No direct comparative evidence was found, and nor was any direct evidence found on clinical outcomes and cost-effectiveness. Artificial intelligence-assisted image analysis may be cost-effective in screening for lung cancer but not for symptomatic populations. However, reliable estimates of cost-effectiveness cannot be obtained with current evidence.

Study registration: This study is registered as PROSPERO CRD42021298449.

Funding: This award was funded by the National Institute for Health and Care Research (NIHR) Evidence Synthesis programme (NIHR award ref: NIHR135325) and is published in full in Health Technology Assessment; Vol. 29, No. 14. See the NIHR Funding and Awards website for further award information.