Designing GTP3 prompts to screen articles for systematic reviews of RCTs

Introduction

Satisfactory sensitivity in screening articles for appropriate inclusion in systematic reviews has not yet been achieved using the group of GPT artificial intelligence (AI) systems. One issue in designing prompts for article screening is that while most of the prompt can be validated before use, i.e. on previously published systematic reviews, the part containing the inclusion criteria cannot. This study aimed to advance work in this area by trying to identify a prompt that is robust to variations in the precise wording of inclusion criteria. Prompts with this property should be able to achieve more consistent performance when applied to similar systematic reviews of health topics.

Methods

A prompt, into which alternative wordings (variants) of inclusion criteria could be inserted, was tested on a training dataset of articles identified during the re-run of electronic searches for a single published review. Modification and re-testing of the prompt was undertaken until satisfactory screening sensitivity across six different inclusion criteria variants was achieved. This prompt was then validated by assessing its performance on three “test” datasets, derived from re-run electronic searches from three different reviews.

Results

A prompt was successfully developed using the training dataset that achieved sensitivities of 95.8 %, 100.0 % & 100.0 % respectively in the three test datasets derived from the three different reviews.

Discussion

Iterative design and testing on inclusion criteria variants produced a prompt that consistently achieved satisfactory screening sensitivity. The classification process was fast, cheap and had high specificity.

Public Interest Summary

Systematic reviews summarise all articles that have tried to answer scientific questions. They are usually the gold standard of evidence in medical science and widely inform healthcare policy. However, they are very expensive and time consuming to write. The initial stage of writing systematic reviews consists of reviewing potentially tens of thousands of scientific abstracts. This process may be able to be automated by artificial intelligence (AI) including GPT3 an AI system operated by OpenAI. Previous attempts to use closely related AI models have not worked likely in part because GPT3´s performance is strongly dependant on the exact instructions or “prompts” given to GPT3. This study investigated a new method of designing these prompts which consistently achieved satisfactory screening performance when tested on articles collected for three previously published systematic reviews.