Comparing Speed and Accuracy of Artificial Intelligence Large Language Models on the Orthopedic In-Training Examination.

Abstract

Objectives: Large language models (LLMs), such as Open AI's Chat Generative Pre-Trained Transformer (GPT)-4 and Google Gemini, have gained significant attention for their ability to process complex language patterns and are being used increasingly in fields such as medicine, where they assist in learning, collaboration, and patient care. Although prior studies have evaluated LLMs on medical licensing examinations, limited research compares their performance on orthopedic-specific assessments. This study aims to assess the accuracy and response speed of ChatGPT-3.5, ChatGPT-4, Microsoft Copilot, and Gemini on the Orthopedic In-Training Examination (OITE).

Methods: Questions from the 2020-2022 OITE were extracted from the American Academy of Orthopaedic Surgeons' question bank. Each question, along with four answer choices, was manually input into the LLMs without response prompts or feedback. Response accuracy and speed were recorded, with timing measured from the moment the question was submitted until an answer was generated.

Results: Out of 1582 prompts, ChatGPT-4 demonstrated the highest accuracy (67.09%±0.08%), significantly outperforming ChatGPT-3.5, Microsoft Copilot, and Gemini (P<0.001). ChatGPT-3.5 was the fastest, with an average response time of 5.41±0.10 seconds. Both ChatGPT-3.5 and ChatGPT-4 responded significantly faster than Gemini and Microsoft Copilot (P<0.001).

Conclusions: ChatGPT-4 exhibited the highest accuracy on OITE questions, and ChatGPT-3.5 was the fastest. Gemini and Copilot were generally less accurate in their responses and had a slower response time. These findings highlight the potential of LLMs in orthopedic education and emphasize the need for further research to explore their broader applications in medical training and decision making.