OpenAI 的新型 o1 模型能否在常见的眼科护理查询中胜过其前辈？

IF 3.2 Q1 OPHTHALMOLOGY

Ophthalmology science Pub Date : 2025-02-22 DOI:10.1016/j.xops.2025.100745

Krithi Pushpanathan MSc , Minjie Zou MMed , Sahana Srinivasan BEng , Wendy Meihua Wong MMed , Erlangga Ariadarma Mangunkusumo MD , George Naveen Thomas MMed , Yien Lai MMed , Chen-Hsin Sun MD , Janice Sing Harn Lam MMed , Marcus Chun Jin Tan MMed , Hazel Anne Hui'En Lin MMed , Weizhi Ma PhD , Victor Teck Chang Koh MMed , David Ziyou Chen MMed , Yih-Chung Tham PhD

{"title":"OpenAI 的新型 o1 模型能否在常见的眼科护理查询中胜过其前辈？","authors":"Krithi Pushpanathan MSc , Minjie Zou MMed , Sahana Srinivasan BEng , Wendy Meihua Wong MMed , Erlangga Ariadarma Mangunkusumo MD , George Naveen Thomas MMed , Yien Lai MMed , Chen-Hsin Sun MD , Janice Sing Harn Lam MMed , Marcus Chun Jin Tan MMed , Hazel Anne Hui'En Lin MMed , Weizhi Ma PhD , Victor Teck Chang Koh MMed , David Ziyou Chen MMed , Yih-Chung Tham PhD","doi":"10.1016/j.xops.2025.100745","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>The newly launched OpenAI o1 is said to offer improved reasoning, potentially providing higher quality responses to eye care queries. However, its performance remains unassessed. We evaluated the performance of o1, ChatGPT-4o, and ChatGPT-4 in addressing ophthalmic-related queries, focusing on correctness, completeness, and readability.</div></div><div><h3>Design</h3><div>Cross-sectional study.</div></div><div><h3>Subjects</h3><div>Sixteen queries, previously identified as suboptimally responded to by ChatGPT-4 from prior studies, were used, covering 3 subtopics: myopia (6 questions), ocular symptoms (4 questions), and retinal conditions (6 questions).</div></div><div><h3>Methods</h3><div>For each subtopic, 3 attending-level ophthalmologists, masked to the model sources, evaluated the responses based on correctness, completeness, and readability (on a 5-point scale for each metric).</div></div><div><h3>Main Outcome Measures</h3><div>Mean summed scores of each model for correctness, completeness, and readability, rated on a 5-point scale (maximum score: 15).</div></div><div><h3>Results</h3><div>O1 scored highest in correctness (12.6) and readability (14.2), outperforming ChatGPT-4, which scored 10.3 (<em>P</em> = 0.010) and 12.4 (<em>P</em> < 0.001), respectively. No significant difference was found between o1 and ChatGPT-4o. When stratified by subtopics, o1 consistently demonstrated superior correctness and readability. In completeness, ChatGPT-4o achieved the highest score of 12.4, followed by o1 (10.8), though the difference was not statistically significant. o1 showed notable limitations in completeness for ocular symptom queries, scoring 5.5 out of 15.</div></div><div><h3>Conclusions</h3><div>While o1 is marketed as offering improved reasoning capabilities, its performance in addressing eye care queries does not significantly differ from its predecessor, ChatGPT-4o. Nevertheless, it surpasses ChatGPT-4, particularly in correctness and readability.</div></div><div><h3>Financial Disclosure(s)</h3><div>Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.</div></div>","PeriodicalId":74363,"journal":{"name":"Ophthalmology science","volume":"5 4","pages":"Article 100745"},"PeriodicalIF":3.2000,"publicationDate":"2025-02-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Can OpenAI's New o1 Model Outperform Its Predecessors in Common Eye Care Queries?\",\"authors\":\"Krithi Pushpanathan MSc , Minjie Zou MMed , Sahana Srinivasan BEng , Wendy Meihua Wong MMed , Erlangga Ariadarma Mangunkusumo MD , George Naveen Thomas MMed , Yien Lai MMed , Chen-Hsin Sun MD , Janice Sing Harn Lam MMed , Marcus Chun Jin Tan MMed , Hazel Anne Hui'En Lin MMed , Weizhi Ma PhD , Victor Teck Chang Koh MMed , David Ziyou Chen MMed , Yih-Chung Tham PhD\",\"doi\":\"10.1016/j.xops.2025.100745\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><h3>Objective</h3><div>The newly launched OpenAI o1 is said to offer improved reasoning, potentially providing higher quality responses to eye care queries. However, its performance remains unassessed. We evaluated the performance of o1, ChatGPT-4o, and ChatGPT-4 in addressing ophthalmic-related queries, focusing on correctness, completeness, and readability.</div></div><div><h3>Design</h3><div>Cross-sectional study.</div></div><div><h3>Subjects</h3><div>Sixteen queries, previously identified as suboptimally responded to by ChatGPT-4 from prior studies, were used, covering 3 subtopics: myopia (6 questions), ocular symptoms (4 questions), and retinal conditions (6 questions).</div></div><div><h3>Methods</h3><div>For each subtopic, 3 attending-level ophthalmologists, masked to the model sources, evaluated the responses based on correctness, completeness, and readability (on a 5-point scale for each metric).</div></div><div><h3>Main Outcome Measures</h3><div>Mean summed scores of each model for correctness, completeness, and readability, rated on a 5-point scale (maximum score: 15).</div></div><div><h3>Results</h3><div>O1 scored highest in correctness (12.6) and readability (14.2), outperforming ChatGPT-4, which scored 10.3 (<em>P</em> = 0.010) and 12.4 (<em>P</em> < 0.001), respectively. No significant difference was found between o1 and ChatGPT-4o. When stratified by subtopics, o1 consistently demonstrated superior correctness and readability. In completeness, ChatGPT-4o achieved the highest score of 12.4, followed by o1 (10.8), though the difference was not statistically significant. o1 showed notable limitations in completeness for ocular symptom queries, scoring 5.5 out of 15.</div></div><div><h3>Conclusions</h3><div>While o1 is marketed as offering improved reasoning capabilities, its performance in addressing eye care queries does not significantly differ from its predecessor, ChatGPT-4o. Nevertheless, it surpasses ChatGPT-4, particularly in correctness and readability.</div></div><div><h3>Financial Disclosure(s)</h3><div>Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.</div></div>\",\"PeriodicalId\":74363,\"journal\":{\"name\":\"Ophthalmology science\",\"volume\":\"5 4\",\"pages\":\"Article 100745\"},\"PeriodicalIF\":3.2000,\"publicationDate\":\"2025-02-22\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Ophthalmology science\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S2666914525000430\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"OPHTHALMOLOGY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Ophthalmology science","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2666914525000430","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"OPHTHALMOLOGY","Score":null,"Total":0}

引用次数: 0

摘要

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Can OpenAI's New o1 Model Outperform Its Predecessors in Common Eye Care Queries?

Objective

The newly launched OpenAI o1 is said to offer improved reasoning, potentially providing higher quality responses to eye care queries. However, its performance remains unassessed. We evaluated the performance of o1, ChatGPT-4o, and ChatGPT-4 in addressing ophthalmic-related queries, focusing on correctness, completeness, and readability.

Design

Cross-sectional study.

Subjects

Sixteen queries, previously identified as suboptimally responded to by ChatGPT-4 from prior studies, were used, covering 3 subtopics: myopia (6 questions), ocular symptoms (4 questions), and retinal conditions (6 questions).

Methods

For each subtopic, 3 attending-level ophthalmologists, masked to the model sources, evaluated the responses based on correctness, completeness, and readability (on a 5-point scale for each metric).

Main Outcome Measures

Mean summed scores of each model for correctness, completeness, and readability, rated on a 5-point scale (maximum score: 15).

Results

O1 scored highest in correctness (12.6) and readability (14.2), outperforming ChatGPT-4, which scored 10.3 (P = 0.010) and 12.4 (P < 0.001), respectively. No significant difference was found between o1 and ChatGPT-4o. When stratified by subtopics, o1 consistently demonstrated superior correctness and readability. In completeness, ChatGPT-4o achieved the highest score of 12.4, followed by o1 (10.8), though the difference was not statistically significant. o1 showed notable limitations in completeness for ocular symptom queries, scoring 5.5 out of 15.

Conclusions

While o1 is marketed as offering improved reasoning capabilities, its performance in addressing eye care queries does not significantly differ from its predecessor, ChatGPT-4o. Nevertheless, it surpasses ChatGPT-4, particularly in correctness and readability.