An AI assistant for critically assessing and synthesizing clusters of journal articles

Abstract

Current large language models (LLMs) face significant challenges in attempting to synthesize and critically assess conflicting causal claims in scientific literature about exposure-associated health effects. This paper examines the design and performance of AIA2, an experimental AI system (freely available at http://cloud.cox-associates.com/) designed to help explore and illustrate potential applications of current AI in assisting analysis of clusters of related scientific articles, focusing on causal claims in complex domains such as epidemiology, toxicology, and risk analysis. Building on an earlier AI assistant, AIA1, which critically reviewed causal claims in individual papers, AIA2 advances the approach by systematically comparing multiple studies to identify areas of agreement and disagreement, suggest explanations for differences in conclusions, flag methodological gaps and inconsistencies, synthesize and summarize well-supported conclusions despite conflicts, and propose recommendations to help resolve knowledge gaps. We illustrate these capabilities with a case study of formaldehyde exposure and leukemia using a cluster of four papers that feature very different approaches and partly conflicting conclusions. AIA2 successfully identifies major points of agreement and contention, discusses the robustness of the evidence for causal claims, and recommends future research directions to address current uncertainties. AIA2's outputs suggest that current AI can offer a promising, practicable approach to AI-assisted review of clusters of papers, promoting methodological rigor, thoroughness, and transparency in review and synthesis, notwithstanding current limitations of LLMs. We discuss the implications of AI-assisted literature review systems for improving evidence-based decision-making, resolving conflicting scientific claims, and promoting rigor and reproducibility in causal research and health risk analysis.