Evaluation of the impact of large language learning models on publications in the Journal of Shoulder and Elbow Surgery

Background

There has been growing interest in the application of artificial intelligence (AI) within academic research to enhance writing, data interpretation, and other tasks. However, it also raises concerns about plagiarism and fraudulent content. While AI-detection tools are available, no systematic review has examined AI use in shoulder and elbow surgery. This study evaluates AI utilization in Journal of Shoulder and Elbow Surgery (JSES) articles before and after the release of ChatGPT (Generative Pre-trained Transformer)-3.5 and explores its correlation with the country of publication.

Methods

We analyzed 232 publications in JSES, from January to April in both 2022 and 2024. Abstracts and full-length texts were manually entered and analyzed using ZeroGPT, an AI-content detector, and ChatGPT detector. A secondary analysis was performed on publications with suspected AI use of greater than 10%, 20%, 30%, 40%, and 50%. Variables analyzed were year of publication, country of origin, and probability of AI use. Univariate analyses according to geographic region were conducted on publications with a suspected AI percentage > 10% and > 20%.

Results

A total of 232 publications were identified, 114 from January to April 2022 (pre-ChatGPT) and 118 from January to April 2024 (post-ChatGPT). The average percentage AI generation was 26% ± 18% within the abstracts and 5% ± 3% within the full-body text of included publications. While there was no significant difference in the suspected use of AI within the full texts before and after the launch of ChatGPT (4.7% ± 3.3% in 2022 vs. 5.3% ± 3.6% in 2024; P = .19), there was a significant increase in the suspected use of AI within the abstracts of articles published after the launch of ChatGPT(21.1% ± 12.8% vs. 30.1% ± 21.6%; P = .0002. Abstracts with a suspected AI percentage > 10% constituted 74.6% of the publications in 2022 and 86.4% in 2024; P = .003. Similarly, abstracts with suspected AI percentage exceeding 20%, 30%, 40%, and 50% also demonstrated statistically significant increases between the 2 periods (P < .05 for all). Univariate analysis revealed that European publications had significantly lower AI content above > 10% (P = .04; odds ratio 0.53, 95% confidence interval: 0.26-0.81).

Conclusion

This study highlights increased use of AI within the writing of JSES publications since the launch of ChatGPT-3.5. While the integration of AI introduces new opportunities in scientific research, there are ethical and methodological challenges that must be carefully considered.