Validating Loon Lens 1.0 for Autonomous Abstract Screening and Confidence-Guided Human-in-the-Loop Workflows in Systematic Reviews.

Objectives: Title and Abstract (TiAb) screening is a labour-intensive step in systematic literature reviews (SLR). We examine the performance of Loon Lens 1.0, an agentic AI platform for autonomous TiAb screening and test whether its confidence scores can target minimal human oversight.

Methods: Eight SLRs by Canada's Drug Agency were re-screened through dual-human reviewers and adjudicated process (3,796 citations, 287 includes, 7.6%) and separately by Loon Lens, based on predefined eligibility criteria. Accuracy, sensitivity, precision, and specificity were measured and bootstrapped to generate 95% confidence intervals. Logistic regression with (i) confidence alone and (ii) confidence + Include/Exclude decision predicted errors and informed simulated human-in-the-loop (HITL) strategies.

Results: Loon Lens achieved 95.5% accuracy (95% CI 94.8-96.1), 98.9% sensitivity (97.6-100), 95.2% specificity (94.5-95.9) and 63.0% precision (58.4-67.3). Errors clustered in Low-Medium-confidence Includes. The extended logistic regression model (confidence + decision; C-index 0.98) estimated a 75% error probability for Low-confidence Includes versus <0.1% for Very-High-confidence Excludes. Simulated HITL review of Low + Medium-confidence Includes only (145 citations, 3.8%), lifted precision to 81.4% and overall accuracy to 98.2% while preserving sensitivity (99.0%). Adding High-confidence Includes (221 citations, 5.8%) pushed precision to 89.9% and accuracy to 99.0%.

Conclusions: Across eight SLRs (3,796 citations), Loon Lens 1.0 reproduced adjudicated human screening with 98.9% sensitivity and 95.2% specificity. In simulation, restricting human-in-the-loop review to ≤5.8% of citations, by prioritising low- and medium-confidence Include calls, reduced false positives and increased precision to 89.9% while maintaining sensitivity and raising overall accuracy to 99.0%. These findings indicate that confidence-guided oversight can concentrate reviewer effort on a small subset of records.