Improving the Reporting Quality of Studies on Information Extraction From Clinical Texts: Protocol for the Development of a Consensus-Based Reporting Guideline.

Abstract

Background: Information extraction (IE) from clinical texts is increasingly important in health care; yet, reporting practices remain inconsistent. Existing guidelines do not fully address the unique challenges of IE studies. IE methods vary widely in their design, ranging from rule-based systems to advanced large language models, contributing to heterogeneity in reporting. While several reporting frameworks exist for applications of artificial intelligence in health care, they primarily focus on prediction modeling or clinical trials and associated protocols rather than text-based IE.

Objective: This study aims to develop the Clinical Information Extraction (CINEX) guideline, a consensus-based reporting guideline for studies on clinical IE.

Methods: The CINEX guideline is developed following an established guideline methodology, including a 3-round electronic Delphi (eDelphi) study with domain experts and a final in-person consensus meeting. The eDelphi process includes feedback loops and predefined consensus thresholds, with items rated on a 10-point scale for both relevance and maturity. The final consensus meeting is held as a hybrid workshop at the MEDINFO 2025 conference and focuses on finalizing the items that reached consensus.

Results: Our results will provide a validated reporting guideline for studies on clinical IE. A preliminary set of 28 reporting items was drafted from a scoping review and existing frameworks. The draft guidelines include 5 key dimensions: information model, architecture, data, annotation, and outcome. This draft guideline will be refined through the eDelphi process. It is designed to be technology-agnostic and applicable across diverse IE approaches, including not only large language models but also traditional machine learning methods and rule-based and hybrid systems.

Conclusions: The CINEX guideline provides structured, expert-validated guidance for reporting clinical IE studies, improving transparency, reproducibility, and comparability. The final guideline will be disseminated alongside an explanatory document to support adoption and implementation.