The Growing Role of Artificial Intelligence in Surgical Education: ChatGPT Undertakes the Australian Generic Surgical Sciences Examination.

Background: Large language models have undergone vast development in recent years. The advent of large language models such as ChatGPT may play an important role in enhancing future medical education.

Methods: To evaluate the accuracy and performance of ChatGPT in the Generic Surgical Sciences Examination, we constructed a sample examination used to assess ChatGPT. Questions were sourced from a past questions bank and formatted to mirror the structure and layout of the examination. The performance of ChatGPT was assessed based on a predefined answer key recorded earlier.

Results: ChatGPT scored a total of 468 marks out of a maximum total of 644 marks, scoring a final percentage of 72.7% across all sections tested. ChatGPT performed best in the physiology section, scoring 77.9%, followed by pathology, scoring 75.0%, and scored lowest in the anatomy section with 66.3%. When scoring was analyzed by question type, it was identified that ChatGPT performed best in the type "A" questions (multiple choice), scoring a total of 75%, which was followed closely by its performance in type "X" questions (true or false), where ChatGPT scored 73.2%. However, ChatGPT only scored 43.8% when answering type "B" questions (establishing a relationship between two statements).

Conclusion: Our results demonstrate that ChatGPT completed the Generic Surgical Sciences Examination with accuracy exceeding the required threshold for a pass in this examination. However, the large language model struggled with certain question types and sections. Overall, further research regarding the utility of ChatGPT in surgical education is required, and caution should be exercised with its use, as it remains in its infancy stages.