EVALUATION OF THE PERFORMANCE OF CHATGPT/ARTIFICIAL INTELLIGENCE IN THE MULTIPLE-CHOICE TEST TO OBTAIN THE TITLE OF SPECIALIST IN ORTHOPEDICS AND TRAUMATOLOGY.

IF 0.5 4区医学 Q4 ORTHOPEDICS

Acta Ortopedica Brasileira Pub Date : 2025-04-07 eCollection Date: 2025-01-01 DOI:10.1590/1413-785220243201e280947

Lucas Plens DE Britto Costa, Danilo Henrique Pizzo DE Castro, Renato Pinheiro Cordeiro, Rômulo Ballarin Albino

{"title":"EVALUATION OF THE PERFORMANCE OF CHATGPT/ARTIFICIAL INTELLIGENCE IN THE MULTIPLE-CHOICE TEST TO OBTAIN THE TITLE OF SPECIALIST IN ORTHOPEDICS AND TRAUMATOLOGY.","authors":"Lucas Plens DE Britto Costa, Danilo Henrique Pizzo DE Castro, Renato Pinheiro Cordeiro, Rômulo Ballarin Albino","doi":"10.1590/1413-785220243201e280947","DOIUrl":null,"url":null,"abstract":"Introduction: ChatGPT, an advanced Artificial Intelligence model specialized in natural language processing, shows remarkable abilities, achieving high scores in certification exams in various specialties. This study aims to evaluate ChatGPT's performance in multiple-choice tests applied to obtain specialist certification in Orthopedics and Traumatology.Methods: We used ChatGPT 4.0 to answer 100 questions from the first phase of the Título de Especialista em Ortopedia e Traumatologia 2022 (TEOT) (Specialist in Orthopedics and Traumatology Test). We excluded non-text-based questions. Each question was entered individually into ChatGPT, with a new session initiated for each question. Performance was evaluated regarding number of words and questions' taxonomic classification.Results: Of the 95 questions analyzed, ChatGPT answered 61.05% correctly and 38.95% incorrectly. There was no statistically significant difference regarding number of words, and ChatGPT's performance did not vary according to taxonomic level.Conclusion: ChatGPT demonstrated vast knowledge in Orthopedics, with acceptable performance in the TEOT exam. Results suggest ChatGPT's an educational and clinical resource in Orthopedics, but needs future progress and human supervision for its effective application. Level of evidence IV, Case series.","PeriodicalId":55563,"journal":{"name":"Acta Ortopedica Brasileira","volume":"33 spe1","pages":"e280947"},"PeriodicalIF":0.5000,"publicationDate":"2025-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11978311/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Acta Ortopedica Brasileira","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1590/1413-785220243201e280947","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/1/1 0:00:00","PubModel":"eCollection","JCR":"Q4","JCRName":"ORTHOPEDICS","Score":null,"Total":0}

引用次数: 0

Abstract

Introduction: ChatGPT, an advanced Artificial Intelligence model specialized in natural language processing, shows remarkable abilities, achieving high scores in certification exams in various specialties. This study aims to evaluate ChatGPT's performance in multiple-choice tests applied to obtain specialist certification in Orthopedics and Traumatology.

Methods: We used ChatGPT 4.0 to answer 100 questions from the first phase of the Título de Especialista em Ortopedia e Traumatologia 2022 (TEOT) (Specialist in Orthopedics and Traumatology Test). We excluded non-text-based questions. Each question was entered individually into ChatGPT, with a new session initiated for each question. Performance was evaluated regarding number of words and questions' taxonomic classification.

Results: Of the 95 questions analyzed, ChatGPT answered 61.05% correctly and 38.95% incorrectly. There was no statistically significant difference regarding number of words, and ChatGPT's performance did not vary according to taxonomic level.

Conclusion: ChatGPT demonstrated vast knowledge in Orthopedics, with acceptable performance in the TEOT exam. Results suggest ChatGPT's an educational and clinical resource in Orthopedics, but needs future progress and human supervision for its effective application. Level of evidence IV, Case series.

查看原文本刊更多论文

评估chatgpt /人工智能在多项选择题中的表现，以获得骨科和创伤科专家的头衔。

简介：ChatGPT是一种专门从事自然语言处理的先进人工智能模型，其能力显著，在各个专业的认证考试中都取得了高分。本研究旨在评估ChatGPT在获得骨科和创伤学专家认证的多项选择题中的表现。方法：我们使用ChatGPT 4.0回答了Título de Especialista em Ortopedia e trauma - ologia 2022 (TEOT) （Specialist in Orthopedics and trauma - ology Test）第一期的100个问题。我们排除了非文本问题。每个问题都单独输入到ChatGPT中，并为每个问题启动一个新会话。根据单词数量和问题的分类分类来评估成绩。结果：在分析的95个问题中，ChatGPT答对61.05%，答错38.95%。在单词数量上没有统计学上的显著差异，ChatGPT的性能也不随分类水平的不同而变化。结论：ChatGPT展示了丰富的骨科知识，在TEOT考试中表现良好。结果表明，ChatGPT是骨科的一种教育和临床资源，但需要进一步的发展和人为的监督才能有效地应用。证据等级IV，案例系列。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Acta Ortopedica Brasileira Orthopedics-

CiteScore

0.90

自引率

14.30%

发文量

审稿时长

25 weeks

期刊介绍： A Revista Acta Ortopédica Brasileira, órgão oficial do Departamento de Ortopedia e Traumatologia da Faculdade de Medicina da Universidade de São Paulo (DOT/FMUSP), é publicada bimestralmente em seis edições ao ano (jan/fev, mar/abr, maio/jun, jul/ago, set/out e nov/dez) com versão em inglês disponível nos principais indexadores nacionais e internacionais e instituições de ensino do Brasil. Sendo hoje reconhecidamente uma importante contribuição para os especialistas da área com sua seriedade e árduo trabalho para as indexações já conquistadas.