Sinan Khadhouri, Artsiom Hramyka, Kevin Gallagher, Alexander Light, Simona Ippoliti, Marie Edison, Cameron Alexander, Meghana Kulkarni, Eleanor Zimmermann, Arjun Nathan, Luca Orecchia, Ravi Banthia, Pietro Piazza, David Mak, Nikolaos Pyrgidis, Prabhat Narayan, Pablo Abad Lopez, Faisal Nawaz, Trung-Thanh Tran, Francesco Claps, Donnacha Hogan, Juan Gomez Rivas, Santiago Alonso, Ijeoma Chibuzo, Beatriz Gutierrez Hidalgo, Jessica Whitburn, Jeremy Teoh, Gautier Marcq, Alexandra Szostek, Jasper Bondad, Petros Sountoulides, Tom Kelsey, Veeru Kasivisvanathan
{"title":"针对因疑似尿路癌转诊至二级医疗机构的血尿患者的 IDENTIFY 风险计算器的机器学习和外部验证。","authors":"Sinan Khadhouri, Artsiom Hramyka, Kevin Gallagher, Alexander Light, Simona Ippoliti, Marie Edison, Cameron Alexander, Meghana Kulkarni, Eleanor Zimmermann, Arjun Nathan, Luca Orecchia, Ravi Banthia, Pietro Piazza, David Mak, Nikolaos Pyrgidis, Prabhat Narayan, Pablo Abad Lopez, Faisal Nawaz, Trung-Thanh Tran, Francesco Claps, Donnacha Hogan, Juan Gomez Rivas, Santiago Alonso, Ijeoma Chibuzo, Beatriz Gutierrez Hidalgo, Jessica Whitburn, Jeremy Teoh, Gautier Marcq, Alexandra Szostek, Jasper Bondad, Petros Sountoulides, Tom Kelsey, Veeru Kasivisvanathan","doi":"10.1016/j.euf.2024.06.004","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>The IDENTIFY study developed a model to predict urinary tract cancer using patient characteristics from a large multicentre, international cohort of patients referred with haematuria. In addition to calculating an individual's cancer risk, it proposes thresholds to stratify them into very-low-risk (<1%), low-risk (1-<5%), intermediate-risk (5-<20%), and high-risk (≥20%) groups.</p><p><strong>Objective: </strong>To externally validate the IDENTIFY haematuria risk calculator and compare traditional regression with machine learning algorithms.</p><p><strong>Design, setting, and participants: </strong>Prospective data were collected on patients referred to secondary care with new haematuria. Data were collected for patient variables included in the IDENTIFY risk calculator, cancer outcome, and TNM staging. Machine learning methods were used to evaluate whether better models than those developed with traditional regression methods existed.</p><p><strong>Outcome measurements and statistical analysis: </strong>The area under the receiver operating characteristic curve (AUC) for the detection of urinary tract cancer, calibration coefficient, calibration in the large (CITL), and Brier score were determined.</p><p><strong>Results and limitations: </strong>There were 3582 patients in the validation cohort. The development and validation cohorts were well matched. The AUC of the IDENTIFY risk calculator on the validation cohort was 0.78. This improved to 0.80 on a subanalysis of urothelial cancer prevalent countries alone, with a calibration slope of 1.04, CITL of 0.24, and Brier score of 0.14. The best machine learning model was Random Forest, which achieved an AUC of 0.76 on the validation cohort. There were no cancers stratified to the very-low-risk group in the validation cohort. Most cancers were stratified to the intermediate- and high-risk groups, with more aggressive cancers in higher-risk groups.</p><p><strong>Conclusions: </strong>The IDENTIFY risk calculator performed well at predicting cancer in patients referred with haematuria on external validation. This tool can be used by urologists to better counsel patients on their cancer risks, to prioritise diagnostic resources on appropriate patients, and to avoid unnecessary invasive procedures in those with a very low risk of cancer.</p><p><strong>Patient summary: </strong>We previously developed a calculator that predicts patients' risk of cancer when they have blood in their urine, based on their personal characteristics. We have validated this risk calculator, by testing it on a separate group of patients to ensure that it works as expected. Most patients found to have cancer tended to be in the higher-risk groups and had more aggressive types of cancer with a higher risk. This tool can be used by clinicians to fast-track high-risk patients based on the calculator and investigate them more thoroughly.</p>","PeriodicalId":12160,"journal":{"name":"European urology focus","volume":" ","pages":"1034-1042"},"PeriodicalIF":4.8000,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Machine Learning and External Validation of the IDENTIFY Risk Calculator for Patients with Haematuria Referred to Secondary Care for Suspected Urinary Tract Cancer.\",\"authors\":\"Sinan Khadhouri, Artsiom Hramyka, Kevin Gallagher, Alexander Light, Simona Ippoliti, Marie Edison, Cameron Alexander, Meghana Kulkarni, Eleanor Zimmermann, Arjun Nathan, Luca Orecchia, Ravi Banthia, Pietro Piazza, David Mak, Nikolaos Pyrgidis, Prabhat Narayan, Pablo Abad Lopez, Faisal Nawaz, Trung-Thanh Tran, Francesco Claps, Donnacha Hogan, Juan Gomez Rivas, Santiago Alonso, Ijeoma Chibuzo, Beatriz Gutierrez Hidalgo, Jessica Whitburn, Jeremy Teoh, Gautier Marcq, Alexandra Szostek, Jasper Bondad, Petros Sountoulides, Tom Kelsey, Veeru Kasivisvanathan\",\"doi\":\"10.1016/j.euf.2024.06.004\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><strong>Background: </strong>The IDENTIFY study developed a model to predict urinary tract cancer using patient characteristics from a large multicentre, international cohort of patients referred with haematuria. In addition to calculating an individual's cancer risk, it proposes thresholds to stratify them into very-low-risk (<1%), low-risk (1-<5%), intermediate-risk (5-<20%), and high-risk (≥20%) groups.</p><p><strong>Objective: </strong>To externally validate the IDENTIFY haematuria risk calculator and compare traditional regression with machine learning algorithms.</p><p><strong>Design, setting, and participants: </strong>Prospective data were collected on patients referred to secondary care with new haematuria. Data were collected for patient variables included in the IDENTIFY risk calculator, cancer outcome, and TNM staging. Machine learning methods were used to evaluate whether better models than those developed with traditional regression methods existed.</p><p><strong>Outcome measurements and statistical analysis: </strong>The area under the receiver operating characteristic curve (AUC) for the detection of urinary tract cancer, calibration coefficient, calibration in the large (CITL), and Brier score were determined.</p><p><strong>Results and limitations: </strong>There were 3582 patients in the validation cohort. The development and validation cohorts were well matched. The AUC of the IDENTIFY risk calculator on the validation cohort was 0.78. This improved to 0.80 on a subanalysis of urothelial cancer prevalent countries alone, with a calibration slope of 1.04, CITL of 0.24, and Brier score of 0.14. The best machine learning model was Random Forest, which achieved an AUC of 0.76 on the validation cohort. There were no cancers stratified to the very-low-risk group in the validation cohort. Most cancers were stratified to the intermediate- and high-risk groups, with more aggressive cancers in higher-risk groups.</p><p><strong>Conclusions: </strong>The IDENTIFY risk calculator performed well at predicting cancer in patients referred with haematuria on external validation. This tool can be used by urologists to better counsel patients on their cancer risks, to prioritise diagnostic resources on appropriate patients, and to avoid unnecessary invasive procedures in those with a very low risk of cancer.</p><p><strong>Patient summary: </strong>We previously developed a calculator that predicts patients' risk of cancer when they have blood in their urine, based on their personal characteristics. We have validated this risk calculator, by testing it on a separate group of patients to ensure that it works as expected. Most patients found to have cancer tended to be in the higher-risk groups and had more aggressive types of cancer with a higher risk. This tool can be used by clinicians to fast-track high-risk patients based on the calculator and investigate them more thoroughly.</p>\",\"PeriodicalId\":12160,\"journal\":{\"name\":\"European urology focus\",\"volume\":\" \",\"pages\":\"1034-1042\"},\"PeriodicalIF\":4.8000,\"publicationDate\":\"2024-12-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"European urology focus\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://doi.org/10.1016/j.euf.2024.06.004\",\"RegionNum\":2,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"2024/6/21 0:00:00\",\"PubModel\":\"Epub\",\"JCR\":\"Q1\",\"JCRName\":\"UROLOGY & NEPHROLOGY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"European urology focus","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.euf.2024.06.004","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/6/21 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"UROLOGY & NEPHROLOGY","Score":null,"Total":0}
Machine Learning and External Validation of the IDENTIFY Risk Calculator for Patients with Haematuria Referred to Secondary Care for Suspected Urinary Tract Cancer.
Background: The IDENTIFY study developed a model to predict urinary tract cancer using patient characteristics from a large multicentre, international cohort of patients referred with haematuria. In addition to calculating an individual's cancer risk, it proposes thresholds to stratify them into very-low-risk (<1%), low-risk (1-<5%), intermediate-risk (5-<20%), and high-risk (≥20%) groups.
Objective: To externally validate the IDENTIFY haematuria risk calculator and compare traditional regression with machine learning algorithms.
Design, setting, and participants: Prospective data were collected on patients referred to secondary care with new haematuria. Data were collected for patient variables included in the IDENTIFY risk calculator, cancer outcome, and TNM staging. Machine learning methods were used to evaluate whether better models than those developed with traditional regression methods existed.
Outcome measurements and statistical analysis: The area under the receiver operating characteristic curve (AUC) for the detection of urinary tract cancer, calibration coefficient, calibration in the large (CITL), and Brier score were determined.
Results and limitations: There were 3582 patients in the validation cohort. The development and validation cohorts were well matched. The AUC of the IDENTIFY risk calculator on the validation cohort was 0.78. This improved to 0.80 on a subanalysis of urothelial cancer prevalent countries alone, with a calibration slope of 1.04, CITL of 0.24, and Brier score of 0.14. The best machine learning model was Random Forest, which achieved an AUC of 0.76 on the validation cohort. There were no cancers stratified to the very-low-risk group in the validation cohort. Most cancers were stratified to the intermediate- and high-risk groups, with more aggressive cancers in higher-risk groups.
Conclusions: The IDENTIFY risk calculator performed well at predicting cancer in patients referred with haematuria on external validation. This tool can be used by urologists to better counsel patients on their cancer risks, to prioritise diagnostic resources on appropriate patients, and to avoid unnecessary invasive procedures in those with a very low risk of cancer.
Patient summary: We previously developed a calculator that predicts patients' risk of cancer when they have blood in their urine, based on their personal characteristics. We have validated this risk calculator, by testing it on a separate group of patients to ensure that it works as expected. Most patients found to have cancer tended to be in the higher-risk groups and had more aggressive types of cancer with a higher risk. This tool can be used by clinicians to fast-track high-risk patients based on the calculator and investigate them more thoroughly.
期刊介绍:
European Urology Focus is a new sister journal to European Urology and an official publication of the European Association of Urology (EAU).
EU Focus will publish original articles, opinion piece editorials and topical reviews on a wide range of urological issues such as oncology, functional urology, reconstructive urology, laparoscopy, robotic surgery, endourology, female urology, andrology, paediatric urology and sexual medicine. The editorial team welcome basic and translational research articles in the field of urological diseases. Authors may be solicited by the Editor directly. All submitted manuscripts will be peer-reviewed by a panel of experts before being considered for publication.