JCO Clinical Cancer Informatics最新文献

{"title":"Artificial Intelligence-Based Digital Histologic Classifier for Prostate Cancer Risk Stratification: Independent Blinded Validation in Patients Treated With Radical Prostatectomy.","authors":"Magdalena Fay, Ross S Liao, Zaeem M Lone, Chandana A Reddy, Hassan Muhammad, Chensu Xie, Parag Jain, Wei Huang, Hirak S Basu, Sujit S Nair, Dimple Chakravarty, Sean R Williamson, Shilpa Gupta, Christopher Weight, Rajat Roy, George Wilding, Ashutosh K Tewari, Eric A Klein, Omar Y Mian","doi":"10.1200/CCI-24-00292","DOIUrl":"10.1200/CCI-24-00292","url":null,"abstract":"<p><strong>Purpose: </strong>Artificial intelligence (AI) tools that identify pathologic features from digitized whole-slide images (WSIs) of prostate cancer (CaP) generate data to predict outcomes. The objective of this study was to evaluate the clinical validity of an AI-enabled prognostic test, PATHOMIQ_PRAD, using a clinical cohort from the Cleveland Clinic.</p><p><strong>Methods: </strong>We conducted a retrospective analysis of PATHOMIQ_PRAD using CaP WSIs from patients who underwent radical prostatectomy (RP) between 2009 and 2022 and did not receive adjuvant therapy. Patients also had Decipher genomic testing available. WSIs were deidentified, anonymized, and outcomes were blinded. Patients were stratified into high-risk and low-risk categories on the basis of predetermined thresholds for PATHOMIQ_PRAD scores (0.45 for biochemical recurrence [BCR] and 0.55 for distant metastasis [DM]).</p><p><strong>Results: </strong>The study included 344 patients who underwent RP with a median follow-up of 4.3 years. Both PathomIQ and Decipher scores were associated with rates of biochemical recurrence-free survival (BCRFS; PathomIQ score >0.45 <i>v</i> ≤0.45, <i>P</i> <.001; Decipher score >0.6 <i>v</i> ≤0.6, <i>P</i> = .002). There were 16 patients who had DM, and 15 were in the high-risk PathomIQ group (Mets Score >0.55). Both PathomIQ and Decipher scores were associated with rates of metastasis-free survival (PathomIQ score >0.55 <i>v</i> ≤0.55, <i>P</i> <.001; Decipher score >0.6 <i>v</i> ≤0.6, <i>P</i> = .0052). Despite the low event rates for metastasis, multivariable regression demonstrated that high PathomIQ score was significantly associated with DM (>0.55 <i>v</i> ≤0.55, hazard ratio, 10.10 [95% CI, 1.28 to 76.92], <i>P</i> = .0284).</p><p><strong>Conclusion: </strong>These findings independently validate PATHOMIQ_PRAD as a reliable predictor of clinical risk in the postprostatectomy setting. PATHOMIQ_PRAD therefore merits prospective evaluation as a risk stratification tool to select patients for adjuvant or early salvage interventions.</p>","PeriodicalId":51626,"journal":{"name":"JCO Clinical Cancer Informatics","volume":"9 ","pages":"e2400292"},"PeriodicalIF":3.3,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12184973/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144327640","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Machine-Learning Algorithms and Treatment Response in Advanced Melanoma.","authors":"Hinpetch Daungsupawong, Viroj Wiwanitkit","doi":"10.1200/CCI-25-00099","DOIUrl":"https://doi.org/10.1200/CCI-25-00099","url":null,"abstract":"","PeriodicalId":51626,"journal":{"name":"JCO Clinical Cancer Informatics","volume":"9 ","pages":"e2500099"},"PeriodicalIF":3.3,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144509320","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Classifying Stereotactic Radiosurgery Patients by Primary Diagnosis Using Natural Language Processing of Clinical Notes.","authors":"Mario Fugal, David Marshall, Alexander V Alekseyenko, Xia Jing, Graham Warren, Jihad Obeid","doi":"10.1200/CCI-24-00268","DOIUrl":"10.1200/CCI-24-00268","url":null,"abstract":"<p><strong>Purpose: </strong>Accurate identification of the primary tumor diagnosis of patients who have undergone stereotactic radiosurgery (SRS) from electronic health records is a critical but challenging task. Traditional methods of identifying the primary tumor histology relying on International Classification of Diseases (ICD)9 and ICD10 CM codes often fall short in granularity and completeness, particularly for patients with metastatic cancer.</p><p><strong>Methods: </strong>In this study, we propose an approach leveraging natural language processing (NLP) algorithms to enhance the accuracy of extracting primary tumor histology from the patient's electronic records.</p><p><strong>Results: </strong>Through manual annotation of patient data and subsequent algorithm training, we achieved improvements in accuracy and efficiency in primary tumor type classification and finding histology subtypes not available in ICD10 CM.</p><p><strong>Conclusion: </strong>Our findings underscore the value of NLP in refining research processes, identifying patients' cohorts, and improving efficiencies with the goal of potentially improving patient outcomes in SRS treatment.</p>","PeriodicalId":51626,"journal":{"name":"JCO Clinical Cancer Informatics","volume":"9 ","pages":"e2400268"},"PeriodicalIF":3.3,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12178166/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144289743","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Patient Perspectives on Technological Barriers and Implementation Strategies Leveraged During a Real-World Remote Symptom Monitoring Program.","authors":"Tanvi V Padalkar, Nicole L Henderson, D'Ambra N Dent, Emma Hendrix, Catherine Smith, Chao-Hui Sylvia Huang, Tara Kaufmann, Chelsea McGowan, Jennifer Young Pierce, Stacey A Ingram, Angela M Stover, Ethan M Basch, Doris Howell, Bryan J Weiner, J Nicholas Odom, Gabrielle B Rocque","doi":"10.1200/CCI-24-00232","DOIUrl":"10.1200/CCI-24-00232","url":null,"abstract":"<p><strong>Purpose: </strong>Remote symptom monitoring (RSM) using electronic patient-reported outcomes leverages digital technologies to gather real-time information on patient experiences for symptom management. This study reports a formative evaluation of technology-related barriers encountered by patients participating in RSM and implementation strategies used to address those barriers in real-world, large-scale RSM implementations.</p><p><strong>Methods: </strong>Purposive sampling was conducted to recruit patients diagnosed with cancer and participating in RSM at the University of Alabama at Birmingham and USA Health Mitchell Cancer Institute for semi-structured interviews. Interviews were coded to identify technology-related barriers using a constant comparative method. Expert Recommendations for Implementing Change list was used to address the barriers to optimize RSM implementation. Barrier-associated themes from the interviews were mapped to implementation strategies.</p><p><strong>Results: </strong>Forty participants age 24-77 years, half of whom were 60 years or older, were interviewed from December 2021 to February 2024. Three barrier themes relevant to technology utilization in RSM were identified: (1) <i>accessibility concerns</i>, (2) <i>digital health literacy</i>, and (3) <i>user interface challenges</i>. Themes were mapped to the implementation strategies as identified by the implementation team. Eight total implementation strategies were used to address these technology barriers: (1) assess for readiness and identify barriers and facilitators, (2) obtain and use patients/consumers and family/caregiver feedback, (3) involve patients/consumers and family members/caregivers, (4) access new funding, (5) change physical structure and equipment, (6) centralize technical assistance, (7) prepare patients/consumers to be active participants, and (8) intervene with patients/consumers to enhance uptake and adherence.</p><p><strong>Conclusion: </strong>Technology-related barriers may limit the uptake of RSM by patients. Addressing these barriers through multimodel assessment and intervention strategies is crucial to ensuring successful implementation of RSM in real-world settings.</p>","PeriodicalId":51626,"journal":{"name":"JCO Clinical Cancer Informatics","volume":"9 ","pages":"e2400232"},"PeriodicalIF":3.3,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12184978/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144337238","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Machine Learning Model Predicts Abnormal Lymphocytosis Associated With Chronic Lymphocytic Leukemia.","authors":"Joseph Aoki, Omar Khalid, Cihan Kaya, Mohamed E Salama","doi":"10.1200/CCI-24-00197","DOIUrl":"10.1200/CCI-24-00197","url":null,"abstract":"<p><strong>Purpose: </strong>The diagnosis of chronic lymphocytic leukemia (CLL) is often delayed several years in advance of disease. Addressing this care gap would aid in identifying at-risk patients who may benefit from targeted evaluation to prevent adverse outcomes. To our knowledge, to date, however, there are no widely utilized machine learning (ML) models that predict development of CLL. Therefore, the objective of this study was to leverage readily available laboratory data to train and test the performance of ML-based risk models for abnormal lymphocytosis associated with CLL.</p><p><strong>Methods: </strong>The observational study population was composed of deidentified laboratory data procured from a large US outpatient network. The 7-year longitudinal data set included 1,090,707 adult patients with the following inclusion criteria: age 50 to 75 years and initial absolute lymphocyte count (ALC) <5 × 10⁹/L. The data set was split into training and held-out test sets, where 80% of the data were used in training and 20% were used for independent testing. ML models were developed using random forest survival methods. The ground truth outcome was abnormal lymphocytosis associated with CLL and monoclonal B-cell lymphocytosis diagnosis: ALC ≥5 × 10⁹/L with ≥40% relative lymphocytosis.</p><p><strong>Results: </strong>The 12-variable risk classifier model accurately predicted ALC ≥5 × 10⁹/L within 5 years and achieved an area under the curve receiver operating characteristic of 0.92. The most important predictors were ALC (initial, slope), WBC (last, max, slope, initial), platelet (last, slope, max, initial), age, and sex.</p><p><strong>Conclusion: </strong>Our ML risk classifier accurately predicts abnormal lymphocytosis associated with CLL using routine laboratory data. Although prospective studies are warranted, the results support the clinical utility of the model to improve timely recognition for patients at a risk of CLL.</p>","PeriodicalId":51626,"journal":{"name":"JCO Clinical Cancer Informatics","volume":"9 ","pages":"e2400197"},"PeriodicalIF":3.3,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12184979/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144486961","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Enhancing Patient-Trial Matching With Large Language Models: A Scoping Review of Emerging Applications and Approaches.","authors":"Hongyu Chen, Xiaohan Li, Xing He, Aokun Chen, James McGill, Emily C Webber, Hua Xu, Mei Liu, Jiang Bian","doi":"10.1200/CCI-25-00071","DOIUrl":"10.1200/CCI-25-00071","url":null,"abstract":"<p><strong>Purpose: </strong>Patient recruitment remains a major bottleneck in clinical trial execution, with inefficient patient-trial matching often causing delays and failures. Recent advancements in large language models (LLMs) offer a promising avenue for automating and improving this process. This scoping review aims to provide a comprehensive synthesis of the emerging applications of LLMs in patient-trial matching.</p><p><strong>Methods: </strong>A comprehensive search was conducted in PubMed, Web of Science, and OpenAlex for literature published between December 1, 2022, and December 31, 2024. Studies were included if they explicitly integrated LLMs into patient-trial matching systems. Data extraction focused on system architectures, patient data processing, eligibility criteria processing, matching techniques, evaluation metrics, and performance.</p><p><strong>Results: </strong>Of the 2,357 studies initially identified, 24 met the inclusion criteria. The majority (21/24) were published in 2024, highlighting the rapid adoption of LLMs in this domain. Most systems used patient-centric matching (17/24), with OpenAI's generative pretrained transformer models being the most commonly used LLM. Core components of these systems included eligibility criteria processing, patient data processing, and matching, with some incorporating retrieval algorithms to enhance computational efficiency. LLM-integrated approaches demonstrated improved accuracy and scalability in patient-trial matching, although challenges such as performance variability, interpretability, and reliance on synthetic data sets remain significant.</p><p><strong>Conclusion: </strong>LLM-based patient-trial matching systems present a transformative opportunity to enhance the efficiency and accuracy of clinical trial recruitment. Despite current limitations related to model generalizability, explainability, and data constraints, future advancements in hybrid modeling strategies, domain-specific fine-tuning, and real-world data set integration could further optimize LLM-based trial matching. Addressing these challenges will be crucial to realizing the full potential of LLMs in streamlining patient recruitment and accelerating clinical trial execution.</p>","PeriodicalId":51626,"journal":{"name":"JCO Clinical Cancer Informatics","volume":"9 ","pages":"e2500071"},"PeriodicalIF":3.3,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12169815/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144259398","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Accuracy and Reproducibility of ChatGPT Responses to Breast Cancer Tumor Board Patients.","authors":"Ning Liao, Cheukfai Li, William J Gradishar, V Suzanne Klimberg, Joshua A Roshal, Taize Yuan, Sanjiv S Agarwala, Vincente K Valero, Sandra M Swain, Julie A Margenthaler, Isabel T Rubio, Sara A Hurvitz, Charles E Geyer, Nancy U Lin, Hope S Rugo, Guochun Zhang, Nanqiu Liu, Charles M Balch","doi":"10.1200/CCI-25-00001","DOIUrl":"https://doi.org/10.1200/CCI-25-00001","url":null,"abstract":"<p><strong>Purpose: </strong>We assessed the accuracy and reproducibility of Chat Generative Pre-Trained Transformer's (ChatGPT) recommendations in response to breast cancer patients by comparing generated outputs with consensus expert opinions.</p><p><strong>Methods: </strong>362 consecutive breast cancer patients sourced from a weekly international breast cancer webinar series were submitted to a tumor board of renowned experts. The same 362 clinical patients were also prompted to ChatGPT-4.0 three separate times to examine reproducibility.</p><p><strong>Results: </strong>Only 46% of ChatGPT-generated content was entirely concordant with the recommendations of breast cancer experts, and only 39% of ChatGPT's responses demonstrated inter-response similarity. ChatGPT's responses demonstrated higher concordance with CEN experts in earlier stages of breast cancer (0, I, II, III) compared to advanced (IV) patients (<i>P</i> = .019). There were less accurate responses from ChatGPT when responding to patients involving molecular markers and genetic testing (<i>P</i> = .025), and in patients involving antibody drug conjugates (<i>P</i> = .006). ChatGPT's responses were not necessarily incorrect but often omitted specific details about clinical management. When the same prompt was independently sent to CEN into the model on three occasions, each time by difference users, ChatGPT's responses exhibited variable content and formatting in 68% (246 out of 362) of patients and were entirely consistent with one another in only 32% of responses.</p><p><strong>Conclusion: </strong>Since this promising clinical decision-making support tool is widely used currently by physicians worldwide, it is important for the user to understand its limitations as currently constructed when responding to multidisciplinary breast cancer patients, and for researchers in the field to continue improving its ability with contemporary, accurate and complete breast cancer information. As currently constructed, ChatGPT is not engineered to generate identical outputs to the same input and was less likely to correctly interpret and recommend treatments for complex breast cancer patients.</p>","PeriodicalId":51626,"journal":{"name":"JCO Clinical Cancer Informatics","volume":"9 ","pages":"e2500001"},"PeriodicalIF":3.3,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144227518","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Deep Learning Model for Natural Language to Assess Effectiveness of Patients With Non-Muscle Invasive Bladder Cancer Receiving Intravesical Bacillus Calmette-Guérin Therapy.","authors":"Makito Miyake, Naohiro Yonemoto, Kanae Togo, Linghua Xu, Tomoyo Oguri, Masayuki Tanaka, Yoshiyuki Hasegawa, Yoshinobu Izawa, Kenji Araki","doi":"10.1200/CCI-24-00249","DOIUrl":"10.1200/CCI-24-00249","url":null,"abstract":"<p><strong>Purpose: </strong>Collecting information on clinical outcomes (recurrence/progression) from complex treatment courses in non-muscle invasive bladder cancer (NMIBC) is challenging and time-consuming. We developed a deep learning natural language processing model to assess outcomes in patients with NMIBC using vast data from electronic health records (EHRs).</p><p><strong>Methods: </strong>This retrospective study analyzed data from Japanese adults with NMIBC who started Bacillus Calmette-Guérin (BCG) induction therapy between April 2016 and June 2022. A Bidirectional Encoder Representations from Transformers (BERT) model was trained to classify outcomes, supported by human review for past history records. The model's performance was assessed by precision, recall, and F1 scores. We compared the effectiveness of BCG therapy between completion (patients who completed therapy) and non-completion groups.</p><p><strong>Results: </strong>Of 372 patients studied, 79.3% and 20.7% were in the completion group and the non-completion group, respectively. The final BERT model achieved average F1 scores of 0.91 and 0.98 for time to recurrence (TTR), and 0.74 and 0.94 for time to progression (TTP) before and after human support, respectively. The hazard ratio for TTR in BCG completion versus non-completion groups was 0.40 (95% CI, 0.26 to 0.62) by a multivariate Cox proportional hazard model and 0.41 (95% CI, 0.26 to 0.63) by inverse probability of treatment weighting.</p><p><strong>Conclusion: </strong>The developed model could compare the clinical outcomes between treatments in patients with NMIBC using EHRs. Human support, although required, was needed in only 10% documents and was deemed feasible. The model was able to demonstrate the difference in TTR and TTP between BCG completion and non-completion groups.</p>","PeriodicalId":51626,"journal":{"name":"JCO Clinical Cancer Informatics","volume":"9 ","pages":"e2400249"},"PeriodicalIF":3.3,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12233173/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144512787","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Open-Source Hybrid Large Language Model Integrated System for Extraction of Breast Cancer Treatment Pathway From Free-Text Clinical Notes.","authors":"Amara Tariq, Madhu Sikha, Allison W Kurian, Kevin Ward, Theresa H M Keegan, Daniel L Rubin, Imon Banerjee","doi":"10.1200/CCI-25-00002","DOIUrl":"10.1200/CCI-25-00002","url":null,"abstract":"<p><strong>Purpose: </strong>Automated curation of breast cancer treatment data with minimal human involvement could accelerate the collection of statewide and nationwide evidence for patient management and assessing the effectiveness of treatment pathways. The primary challenges are the complexity and inconsistency of structured clinical data streams and accurate extraction of this information from free-text clinical narratives.</p><p><strong>Materials and methods: </strong>We proposed a hybrid two-phase information extraction framework that combined a Unified Medical Language System parser (phase-1) with a fine-tuned large language model (LLM; phase-2) to extract longitudinal treatment timelines from time-stamped clinical notes. Our framework was developed through end-to-end joint learning as a question-answering model, where the model was trained to simultaneously answer five questions, each corresponding to a specific treatment.</p><p><strong>Results: </strong>We fine-tuned and internally validated the model on 26,692 patients with breast cancer (diagnosed between 2013 and 2020) receiving treatment at Mayo Clinic and externally validated the model on 162 randomly selected patients from Stanford Healthcare. Zero-shot LLM (out-of-the-box) had high specificity but low sensitivity, indicating that although these frameworks are useful for generic language understanding, they are lacking in terms of targeted clinical tasks. The proposed model achieved 0.942 average AUROC on the internal and 0.924 on the external data, demonstrating only marginal drop in performance when evaluated on external. The proposed model also achieved better trade-off between sensitivity (average: 79.2%) and specificity (average: 76.2%) compared with rule-based (average sensitivity: 70.5%, average specificity: 68.1%) and structured codes (average sensitivity: 64.1%, average specificity: 83.5%).</p><p><strong>Conclusion: </strong>The proposed framework can extract temporal information about cancer treatments from various time-stamped clinic notes, regardless of the setting of treatment administration (inpatient or outpatient) or time frame. To support the cancer research community for such data curation and longitudinal analysis, we have packaged the code as a docker image, which needs minimal system reconfiguration and shared with an open-source academic license.</p>","PeriodicalId":51626,"journal":{"name":"JCO Clinical Cancer Informatics","volume":"9 ","pages":"e2500002"},"PeriodicalIF":3.3,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12208650/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144512802","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Clinical Trial Design Approach to Auditing Language Models in Health Care Setting.","authors":"Lovedeep Gondara, Jonathan Simkin, Shebnum Devji","doi":"10.1200/CCI-24-00331","DOIUrl":"https://doi.org/10.1200/CCI-24-00331","url":null,"abstract":"<p><strong>Purpose: </strong>Rapid advancements in natural language processing have led to the development of sophisticated language models. Inspired by their success, these models are now used in health care for tasks such as clinical documentation and medical record classification. However, language models are prone to errors, which can have serious consequences in critical domains such as health care, ensuring that their reliability is essential to maintain patient safety and data integrity.</p><p><strong>Methods: </strong>To address this, we propose an innovative auditing process based on principles from clinical trial design. Our approach involves subject matter experts (SMEs) manually reviewing pathology reports without previous knowledge of the model's classification. This single-blind setup minimizes bias and allows us to apply statistical rigor to assess model performance.</p><p><strong>Results: </strong>Deployed at the British Columbia Cancer Registry, our audit process effectively identified the core issues in the operational models. Early interventions addressed these issues, maintaining data integrity and patient care standards.</p><p><strong>Conclusion: </strong>The audit provides real-world performance metrics and underscores the importance of human-in-the-loop machine learning. Even advanced models require SME oversight to ensure accuracy and reliability. To our knowledge, we have developed the first continuous audit process for language models in health care, modeled after clinical trial principles. This methodology ensures that audits are statistically sound and operationally feasible, setting a new standard for evaluating language models in critical applications.</p>","PeriodicalId":51626,"journal":{"name":"JCO Clinical Cancer Informatics","volume":"9 ","pages":"e2400331"},"PeriodicalIF":3.3,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144217490","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}