Generative AI in peer review process for occupational health.

Abstract

Background: Generative Artificial Intelligence (AI) tools in academic writing can augment and speed up the proofing process by improving sections of the manuscript. This was the first known instance where the effectiveness and efficiency of Generative AI were quantified.

Aims: To determine the effectiveness and efficiency of these tools in providing feedback and recommendations to the first drafts of eight published occupational health papers.

Methods: To assess effectiveness, manuscripts were reviewed by Microsoft Copilot, ChatGPT (GPT-3.5), Google Gemini 1.0 and five human reviewers in February 2024. Anonymized reviews were scored by two expert panel members using a self-developed structured scoring system. The quality of feedback was rated on its relevance, completeness, accuracy and ability to identify errors and provide constructive feedback. The quality of recommendations was rated on relevance, completeness and accuracy. Efficiency was assessed via the time taken to complete each review. The mean, standard deviation (SD) and level of significance of any differences among the parameters were obtained.

Results: Generative AI tools were significantly more effective (3.44, SD 0.77, P < 0.001) than human reviewers in providing feedback, while human reviewers performed significantly better (3.36, SD 0.71, P < 0.01) in providing recommendations. Generative AI tools were significantly more time-efficient and had the advantage of being always available. However, time/effort was required to verify the output for fictitious content.

Conclusions: The utilization of Generative AI would improve the speed and accuracy of improving the manuscript prior to publication, leading to greater efficiencies in the dissemination of knowledge to the occupational health community.