Lennart Walger, Tobias Bauer, David Kügler, Matthias H Schmitz, Fabiane Schuch, Christophe Arendt, Tobias Baumgartner, Johannes Birkenheier, Valeri Borger, Christoph Endler, Franziska Grau, Christian Immanuel, Markus Kölle, Patrick Kupczyk, Asadeh Lakghomi, Sarah Mackert, Elisabeth Neuhaus, Julia Nordsiek, Anna-Maria Odenthal, Karmele Olaciregui Dague, Laura Ostermann, Jan Pukropski, Attila Racz, Klaus von der Ropp, Frederic Carsten Schmeel, Felix Schrader, Aileen Sitter, Alexander Unruh-Pinheiro, Marilia Voigt, Martin Vychopen, Philip von Wedel, Randi von Wrede, Ulrike Attenberger, Hartmut Vatter, Alexandra Philipsen, Albert Becker, Martin Reuter, Elke Hattingen, Josemir W Sander, Alexander Radbruch, Rainer Surges, Theodor Rüber
{"title":"人类与人工智能在检测局灶性皮质发育不良方面的定量比较。","authors":"Lennart Walger, Tobias Bauer, David Kügler, Matthias H Schmitz, Fabiane Schuch, Christophe Arendt, Tobias Baumgartner, Johannes Birkenheier, Valeri Borger, Christoph Endler, Franziska Grau, Christian Immanuel, Markus Kölle, Patrick Kupczyk, Asadeh Lakghomi, Sarah Mackert, Elisabeth Neuhaus, Julia Nordsiek, Anna-Maria Odenthal, Karmele Olaciregui Dague, Laura Ostermann, Jan Pukropski, Attila Racz, Klaus von der Ropp, Frederic Carsten Schmeel, Felix Schrader, Aileen Sitter, Alexander Unruh-Pinheiro, Marilia Voigt, Martin Vychopen, Philip von Wedel, Randi von Wrede, Ulrike Attenberger, Hartmut Vatter, Alexandra Philipsen, Albert Becker, Martin Reuter, Elke Hattingen, Josemir W Sander, Alexander Radbruch, Rainer Surges, Theodor Rüber","doi":"10.1097/RLI.0000000000001125","DOIUrl":null,"url":null,"abstract":"<p><strong>Objectives: </strong>Artificial intelligence (AI) is thought to improve lesion detection. However, a lack of knowledge about human performance prevents a comparative evaluation of AI and an accurate assessment of its impact on clinical decision-making. The objective of this work is to quantitatively evaluate the ability of humans to detect focal cortical dysplasia (FCD), compare it to state-of-the-art AI, and determine how it may aid diagnostics.</p><p><strong>Materials and methods: </strong>We prospectively recorded the performance of readers in detecting FCDs using single points and 3-dimensional bounding boxes. We acquired predictions of 3 AI models for the same dataset and compared these to readers. Finally, we analyzed pairwise combinations of readers and models.</p><p><strong>Results: </strong>Twenty-eight readers, including 20 nonexpert and 5 expert physicians, reviewed 180 cases: 146 subjects with FCD (median age: 25, interquartile range: 18) and 34 healthy control subjects (median age: 43, interquartile range: 19). Nonexpert readers detected 47% (95% confidence interval [CI]: 46, 49) of FCDs, whereas experts detected 68% (95% CI: 65, 71). The 3 AI models detected 32%, 51%, and 72% of FCDs, respectively. The latter, however, also predicted more than 13 false-positive clusters per subject on average. Human performance was improved in the presence of a transmantle sign ( P < 0.001) and cortical thickening ( P < 0.001). In contrast, AI models were sensitive to abnormal gyration ( P < 0.01) or gray-white matter blurring ( P < 0.01). Compared with single experts, expert-expert pairs detected 13% (95% CI: 9, 18) more FCDs ( P < 0.001). All AI models increased expert detection rates by up to 19% (95% CI: 15, 24) ( P < 0.001). Nonexpert+AI pairs could still outperform single experts by up to 13% (95% CI: 10, 17).</p><p><strong>Conclusions: </strong>This study pioneers the comparative evaluation of humans and AI for FCD lesion detection. It shows that AI and human predictions differ, especially for certain MRI features of FCD, and, thus, how AI may complement the diagnostic workup.</p>","PeriodicalId":14486,"journal":{"name":"Investigative Radiology","volume":" ","pages":""},"PeriodicalIF":7.0000,"publicationDate":"2024-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"A Quantitative Comparison Between Human and Artificial Intelligence in the Detection of Focal Cortical Dysplasia.\",\"authors\":\"Lennart Walger, Tobias Bauer, David Kügler, Matthias H Schmitz, Fabiane Schuch, Christophe Arendt, Tobias Baumgartner, Johannes Birkenheier, Valeri Borger, Christoph Endler, Franziska Grau, Christian Immanuel, Markus Kölle, Patrick Kupczyk, Asadeh Lakghomi, Sarah Mackert, Elisabeth Neuhaus, Julia Nordsiek, Anna-Maria Odenthal, Karmele Olaciregui Dague, Laura Ostermann, Jan Pukropski, Attila Racz, Klaus von der Ropp, Frederic Carsten Schmeel, Felix Schrader, Aileen Sitter, Alexander Unruh-Pinheiro, Marilia Voigt, Martin Vychopen, Philip von Wedel, Randi von Wrede, Ulrike Attenberger, Hartmut Vatter, Alexandra Philipsen, Albert Becker, Martin Reuter, Elke Hattingen, Josemir W Sander, Alexander Radbruch, Rainer Surges, Theodor Rüber\",\"doi\":\"10.1097/RLI.0000000000001125\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><strong>Objectives: </strong>Artificial intelligence (AI) is thought to improve lesion detection. However, a lack of knowledge about human performance prevents a comparative evaluation of AI and an accurate assessment of its impact on clinical decision-making. The objective of this work is to quantitatively evaluate the ability of humans to detect focal cortical dysplasia (FCD), compare it to state-of-the-art AI, and determine how it may aid diagnostics.</p><p><strong>Materials and methods: </strong>We prospectively recorded the performance of readers in detecting FCDs using single points and 3-dimensional bounding boxes. We acquired predictions of 3 AI models for the same dataset and compared these to readers. Finally, we analyzed pairwise combinations of readers and models.</p><p><strong>Results: </strong>Twenty-eight readers, including 20 nonexpert and 5 expert physicians, reviewed 180 cases: 146 subjects with FCD (median age: 25, interquartile range: 18) and 34 healthy control subjects (median age: 43, interquartile range: 19). Nonexpert readers detected 47% (95% confidence interval [CI]: 46, 49) of FCDs, whereas experts detected 68% (95% CI: 65, 71). The 3 AI models detected 32%, 51%, and 72% of FCDs, respectively. The latter, however, also predicted more than 13 false-positive clusters per subject on average. Human performance was improved in the presence of a transmantle sign ( P < 0.001) and cortical thickening ( P < 0.001). In contrast, AI models were sensitive to abnormal gyration ( P < 0.01) or gray-white matter blurring ( P < 0.01). Compared with single experts, expert-expert pairs detected 13% (95% CI: 9, 18) more FCDs ( P < 0.001). All AI models increased expert detection rates by up to 19% (95% CI: 15, 24) ( P < 0.001). Nonexpert+AI pairs could still outperform single experts by up to 13% (95% CI: 10, 17).</p><p><strong>Conclusions: </strong>This study pioneers the comparative evaluation of humans and AI for FCD lesion detection. It shows that AI and human predictions differ, especially for certain MRI features of FCD, and, thus, how AI may complement the diagnostic workup.</p>\",\"PeriodicalId\":14486,\"journal\":{\"name\":\"Investigative Radiology\",\"volume\":\" \",\"pages\":\"\"},\"PeriodicalIF\":7.0000,\"publicationDate\":\"2024-11-12\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Investigative Radiology\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://doi.org/10.1097/RLI.0000000000001125\",\"RegionNum\":1,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Investigative Radiology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1097/RLI.0000000000001125","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING","Score":null,"Total":0}
A Quantitative Comparison Between Human and Artificial Intelligence in the Detection of Focal Cortical Dysplasia.
Objectives: Artificial intelligence (AI) is thought to improve lesion detection. However, a lack of knowledge about human performance prevents a comparative evaluation of AI and an accurate assessment of its impact on clinical decision-making. The objective of this work is to quantitatively evaluate the ability of humans to detect focal cortical dysplasia (FCD), compare it to state-of-the-art AI, and determine how it may aid diagnostics.
Materials and methods: We prospectively recorded the performance of readers in detecting FCDs using single points and 3-dimensional bounding boxes. We acquired predictions of 3 AI models for the same dataset and compared these to readers. Finally, we analyzed pairwise combinations of readers and models.
Results: Twenty-eight readers, including 20 nonexpert and 5 expert physicians, reviewed 180 cases: 146 subjects with FCD (median age: 25, interquartile range: 18) and 34 healthy control subjects (median age: 43, interquartile range: 19). Nonexpert readers detected 47% (95% confidence interval [CI]: 46, 49) of FCDs, whereas experts detected 68% (95% CI: 65, 71). The 3 AI models detected 32%, 51%, and 72% of FCDs, respectively. The latter, however, also predicted more than 13 false-positive clusters per subject on average. Human performance was improved in the presence of a transmantle sign ( P < 0.001) and cortical thickening ( P < 0.001). In contrast, AI models were sensitive to abnormal gyration ( P < 0.01) or gray-white matter blurring ( P < 0.01). Compared with single experts, expert-expert pairs detected 13% (95% CI: 9, 18) more FCDs ( P < 0.001). All AI models increased expert detection rates by up to 19% (95% CI: 15, 24) ( P < 0.001). Nonexpert+AI pairs could still outperform single experts by up to 13% (95% CI: 10, 17).
Conclusions: This study pioneers the comparative evaluation of humans and AI for FCD lesion detection. It shows that AI and human predictions differ, especially for certain MRI features of FCD, and, thus, how AI may complement the diagnostic workup.
期刊介绍：
Investigative Radiology publishes original, peer-reviewed reports on clinical and laboratory investigations in diagnostic imaging, the diagnostic use of radioactive isotopes, computed tomography, positron emission tomography, magnetic resonance imaging, ultrasound, digital subtraction angiography, and related modalities. Emphasis is on early and timely publication. Primarily research-oriented, the journal also includes a wide variety of features of interest to clinical radiologists.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
