Leveraging generative artificial intelligence for the development of non-interventional research study protocols: a proof-of-concept feasibility study.

Abstract

Background: The writing of study protocols is a labor and time-intensive process. We hypothesized that the writing of some methodological aspects of study protocols for non-interventional studies (NIS) could be amenable to automation using generative artificial intelligence (GenAI), particularly when provided with standardized templates, and a set of common elements, such as study design and objectives.

Methods: In this proof-of-concept feasibility study, we explored whether a large language model (LLM), specifically GPT-4, could support the drafting of selected methodological sections of a study protocol for retrospective observational NIS. The sections to be populated were the objectives, study design, study population, data source, data collection methods, statistical analysis methods and study strengths and limitations. A Python application programming interface (API) was used to send instructions, "prompts", to GPT-4 including guideline instructions, examples of text and specific study inputs related to the research question, including study population, objectives, and data source/data collection methods. The LLM-based program was first tested using four case studies. The program was then revised and refined using an additional four previously unseen case studies, to explore whether the prompting framework could be applied across different objectives, populations and datasets within the defined scope of retrospective observational NIS (quality assessment set, wave 1). Finally, the revised program was applied to a further five previously unseen protocols as a final feasibility assessment (quality assessment set, wave 2). A critical appraisal of the nine protocols populated by GPT-4 in the quality assessment stages was conducted to explore the alignment of the GPT-4 text in relation to the original human-written protocols and against standard guidelines. In addition, two GPT-4 written protocols were blind-reviewed (i.e. "author" unknown) through the Company's routine internal scientific review process.

Results: The critical appraisal of the nine GPT-4-produced protocols from the quality assessment sets suggested that the GPT-4 text aligned well with both the original human produced content and with content required by guidelines. This was further demonstrated by the two blind-reviewed protocols which were approved with only minor comments.

Conclusions: Considerable effort and time (~ 3-6 months) were required to develop the structured prompt engineering workflow with prompts that were transferable to other NIS protocols. When provided with integral details to specify the elements of the study question (via human input), GPT-4 showed promising alignment with human-authored reference protocols under a constrained use case. Critical human input remains essential to define the study question, provide structured inputs, and review and revise the generated text as is current practice for human-written protocols.