Evaluating generative AI models for explainable pathological feature extraction in lung adenocarcinoma grading assessment and prognostic model construction.

Abstract

Background: Given the increasing prevalence of generative AI (GenAI) models, a systematically evaluation of their performance in lung adenocarcinoma histopathological assessment is crucial. This study aimed to evaluate and compare three visual-capable GenAI models (GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro) for lung adenocarcinoma histological pattern recognition and grading, as well as to explore prognostic prediction models based on GenAI feature extraction.

Materials and methods: In this retrospective study, we analyzed 310 diagnostic slides from The Cancer Genome Atlas Lung Adenocarcinoma (TCGA-LUAD) database to evaluate GenAI models and to develop and internally validate machine learning-based prognostic models. For independent external validation, we utilized 95 and 87 slides from obtained different institutions. The primary endpoints comprised GenAI grading accuracy (area under the receiver operating characteristic curve, AUC) and stability (intraclass correlation coefficient, ICC). Secondary endpoints included developing and assessing machine learning-based prognostic models using GenAI-extracted features from the TCGA-LUAD dataset, evaluated by Concordance index (C-index).

Results: Among the evaluated models, claude-3.5-Sonnet demonstrated the best overall performance, achieving high grading accuracy (average AUC = 0.823) with moderate stability (ICC = 0.585) The optimal machine learning-based prognostic model, developed using features extracted by Claude-3.5-Sonnet and integrating clinical variables, demonstrated good performance in both internal and external validations, yielding an average C-index of 0.715. Meta-analysis demonstrated that this prognostic model effectively stratified patients into risk groups, with the high-risk group showing significantly worse outcomes (Hazard ratio = 5.16, 95% confidence interval = 3.09-8.62).

Conclusion: GenAI models demonstrated significant potential in lung adenocarcinoma pathology, with Claude-3.5-Sonnet exhibiting superior performance in grading prediction and robust prognostic capabilities. These findings indicate promising applications of AI in lung adenocarcinoma diagnosis and clinical management.