Epidemiologic Method Review at Scale: Assessing Charlson Comorbidity Versioning Using a LLM.

Abstract

Background: The Charlson Comorbidity Index (CCI) is widely used in epidemiologic studies. However, many versions of the CCI have been developed since the original method was published in 1987, and it is unclear which version is used most frequently and how version utilization in research has changed over time.

Objective: We present an approach using a Large Language Model (LLM) to extract data from articles by detecting which specific CCI version is employed.

Design: We designed a series of prompts that carefully guided the LLM through the identification and extraction of references used in the calculation of the CCI for each particular article. We used the Llama 3.3 70B Instruct model to identify and extract which references were used in the calculation of the CCI.

Setting: We analyzed 31,767 articles published since 2012 to evaluate the landscape of CCI implementation. The articles were sourced from the PubMed Central Open Access subset.

Measurements: For each article, we measured which version of the CCI was used, if any.

Results: We show that 63% of articles that cite only a single method version cite only the original CCI publication, which cannot be applied in the modern real-world data era, leading to ambiguity about how the CCI is being calculated.

Limitations: For articles that did not reference one of the CCI versions we searched for, we were unable to determine whether the paper used a different version, created a specific implementation for that paper, or is ambiguous about how the CCI was calculated.

Conclusion: This paper introduces a generalizable approach to scaling methods literature review beyond what is typically possible by human-review, which we then validate and demonstrate the value of through application to the CCI.