Development and validation of a natural language processing system to assess quality of physician communication in prostate cancer consultations.

IF 5.8 2区医学 Q1 ONCOLOGY

Prostate Cancer and Prostatic Diseases Pub Date : 2025-08-21 DOI:10.1038/s41391-025-01011-5

Renning Zheng, Nadine A Friedrich, Michael Luu, Rebecca Gale, Dmitry Khodyakov, Stephen J Freedland, Brennan Spiegel, Timothy J Daskivich

{"title":"Development and validation of a natural language processing system to assess quality of physician communication in prostate cancer consultations.","authors":"Renning Zheng, Nadine A Friedrich, Michael Luu, Rebecca Gale, Dmitry Khodyakov, Stephen J Freedland, Brennan Spiegel, Timothy J Daskivich","doi":"10.1038/s41391-025-01011-5","DOIUrl":null,"url":null,"abstract":"Background: AUA guidelines for shared decision making (SDM) in prostate cancer recommend discussion of five content areas in consultations: (1) cancer severity (tumor risk (TR), pathology results (PR)); (2) life expectancy (LE); (3) cancer prognosis (CP); (4) baseline urinary and erectile function (UF and EF); and (5) treatment side effects (erectile dysfunction (ED), urinary incontinence (UI), and irritative urinary symptoms (LUTS)). However, patient retention of information after the visit and inconsistent risk communication by physicians are barriers to informed SDM. We sought to develop natural language processing (NLP) models based on recorded consultations to provide key information to patients and audit quality of physician communication.Methods: We used 50 consultation transcripts to train and validate NLP models to identify sentences related to key concepts. We then tested whether communication quality across entire consultations could be determined by sentences with the highest model-predicted topic concordance in 20 separate consultation transcripts.Results: Our development dataset included 28,927 total sentences, with 75% reserved for training and 25% for internal validation. The Random Forest model had the highest accuracy for identifying topic-concordant sentences, with area under the curve 0.98, 0.94, 0.89, 0.92, 0.84, 0.96, 0.98, 0.97, and 0.99 for TR, PR, LE, CP, UF, EF, ED, UI, and LUTS compared with manual coding across all concepts in the internal validation dataset. In 20 separate consultations, the top 10 model-identified sentences correctly graded communication quality across entire consultations with accuracies of 100%, 90%, 95%, 95%, 80%, 95%, 85%, 100%, and 95% for TR, PR, LE, CP, UF, EF, ED, UI, and LUTS compared with manual coding, respectively.Conclusions: NLP models accurately capture key information and grade quality of physician communication in prostate cancer consultations, providing the foundation for scalable quality assessment of risk communication.","PeriodicalId":20727,"journal":{"name":"Prostate Cancer and Prostatic Diseases","volume":" ","pages":""},"PeriodicalIF":5.8000,"publicationDate":"2025-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Prostate Cancer and Prostatic Diseases","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1038/s41391-025-01011-5","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ONCOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Background: AUA guidelines for shared decision making (SDM) in prostate cancer recommend discussion of five content areas in consultations: (1) cancer severity (tumor risk (TR), pathology results (PR)); (2) life expectancy (LE); (3) cancer prognosis (CP); (4) baseline urinary and erectile function (UF and EF); and (5) treatment side effects (erectile dysfunction (ED), urinary incontinence (UI), and irritative urinary symptoms (LUTS)). However, patient retention of information after the visit and inconsistent risk communication by physicians are barriers to informed SDM. We sought to develop natural language processing (NLP) models based on recorded consultations to provide key information to patients and audit quality of physician communication.

Methods: We used 50 consultation transcripts to train and validate NLP models to identify sentences related to key concepts. We then tested whether communication quality across entire consultations could be determined by sentences with the highest model-predicted topic concordance in 20 separate consultation transcripts.

Results: Our development dataset included 28,927 total sentences, with 75% reserved for training and 25% for internal validation. The Random Forest model had the highest accuracy for identifying topic-concordant sentences, with area under the curve 0.98, 0.94, 0.89, 0.92, 0.84, 0.96, 0.98, 0.97, and 0.99 for TR, PR, LE, CP, UF, EF, ED, UI, and LUTS compared with manual coding across all concepts in the internal validation dataset. In 20 separate consultations, the top 10 model-identified sentences correctly graded communication quality across entire consultations with accuracies of 100%, 90%, 95%, 95%, 80%, 95%, 85%, 100%, and 95% for TR, PR, LE, CP, UF, EF, ED, UI, and LUTS compared with manual coding, respectively.

Conclusions: NLP models accurately capture key information and grade quality of physician communication in prostate cancer consultations, providing the foundation for scalable quality assessment of risk communication.

查看原文本刊更多论文

自然语言处理系统的开发和验证，以评估前列腺癌会诊中医生沟通的质量。

背景：AUA前列腺癌共同决策（SDM）指南建议在会诊中讨论五个内容领域：(1)癌症严重程度（肿瘤风险（TR），病理结果（PR））；(2)预期寿命（LE）；(3)肿瘤预后（CP）；(4)基线泌尿和勃起功能（UF和EF）；(5)治疗副作用（勃起功能障碍（ED）、尿失禁（UI）和刺激性尿症状（LUTS））。然而，患者在访问后保留信息和医生不一致的风险沟通是知情SDM的障碍。我们试图开发基于会诊记录的自然语言处理（NLP）模型，为患者提供关键信息并审核医生沟通的质量。方法：我们使用50份咨询记录来训练和验证NLP模型，以识别与关键概念相关的句子。然后，我们测试了整个咨询的沟通质量是否可以由20个单独的咨询记录中具有最高模型预测主题一致性的句子来决定。结果：我们的开发数据集包括28,927个句子，其中75%用于训练，25%用于内部验证。随机森林模型在识别主题一致句子方面的准确率最高，与人工编码相比，TR、PR、LE、CP、UF、EF、ED、UI和LUTS的曲线下面积分别为0.98、0.94、0.89、0.92、0.84、0.96、0.98、0.97和0.99。在20个单独的咨询中，与手动编码相比，前10个模型识别的句子在整个咨询中正确地对通信质量进行了评分，TR、PR、LE、CP、UF、EF、ED、UI和LUTS的准确率分别为100%、90%、95%、95%、95%、80%、95%、85%、100%和95%。结论：NLP模型准确捕获了前列腺癌会诊中医生沟通的关键信息和质量等级，为可扩展的风险沟通质量评估提供了基础。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Prostate Cancer and Prostatic Diseases 医学-泌尿学与肾脏学

CiteScore

10.00

自引率

6.20%

发文量

142

审稿时长

6-12 weeks

期刊介绍： Prostate Cancer and Prostatic Diseases covers all aspects of prostatic diseases, in particular prostate cancer, the subject of intensive basic and clinical research world-wide. The journal also reports on exciting new developments being made in diagnosis, surgery, radiotherapy, drug discovery and medical management. Prostate Cancer and Prostatic Diseases is of interest to surgeons, oncologists and clinicians treating patients and to those involved in research into diseases of the prostate. The journal covers the three main areas - prostate cancer, male LUTS and prostatitis. Prostate Cancer and Prostatic Diseases publishes original research articles, reviews, topical comment and critical appraisals of scientific meetings and the latest books. The journal also contains a calendar of forthcoming scientific meetings. The Editors and a distinguished Editorial Board ensure that submitted articles receive fast and efficient attention and are refereed to the highest possible scientific standard. A fast track system is available for topical articles of particular significance.