Automated classification of online reviews of otolaryngologists

IF 1.6 4区医学 Q2 OTORHINOLARYNGOLOGY

Laryngoscope Investigative Otolaryngology Pub Date : 2024-11-13 DOI:10.1002/lio2.70036

Jake G. Stenzel MS, Nicholas R. Schultz MS, Michael J. Marino MD

{"title":"Automated classification of online reviews of otolaryngologists","authors":"Jake G. Stenzel MS, Nicholas R. Schultz MS, Michael J. Marino MD","doi":"10.1002/lio2.70036","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Objectives</h3>\n \n <p>The study aimed to extract online comments of otolaryngologists in the 20 most populated cities in the United States from healthgrades.com, develop and validate a natural language processing (NLP) logistic regression algorithm for automated text classification of reviews into 10 categories, and compare 1- and 5-star reviews in directly-physician-related and non-physician-related categories.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>1977 1-star and 12,682 5-star reviews were collected. The primary investigator manually categorized a training dataset of 324 1-star and 909 5-star reviews, while a validation subset of 100 5-star and 50 1-star reviews underwent dual manual categorization. Using scikit-learn, an NLP algorithm was trained and validated on the subsets, with F1 scores evaluating text classification accuracy against manual categorization. The algorithm was then applied to the entire dataset with comparison of review categorization according to 1- and 5-star reviews.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>F1 scores for NLP validation ranged between 0.71 and 0.97. Significant associations emerged between 1-star reviews and treatment plan, accessibility, wait time, office scheduling, billing, and facilities. 5-star reviews were associated with surgery/procedure, bedside manner, and staff/mid-levels.</p>\n </section>\n \n <section>\n \n <h3> Conclusion</h3>\n \n <p>The study successfully validated an NLP text classification system for categorizing online physician reviews. Positive reviews were found to have an association with directly-physician related context. 1-star reviews were related to treatment plan, accessibility, wait time, office scheduling, billing, and facilities. This method of text classification effectively discerned the nuances of human-written text, providing valuable insights into online healthcare feedback that is scalable.</p>\n \n <p>Level of evidence: Level 3</p>\n </section>\n </div>","PeriodicalId":48529,"journal":{"name":"Laryngoscope Investigative Otolaryngology","volume":"9 6","pages":""},"PeriodicalIF":1.6000,"publicationDate":"2024-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11558699/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Laryngoscope Investigative Otolaryngology","FirstCategoryId":"3","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/lio2.70036","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"OTORHINOLARYNGOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Objectives

The study aimed to extract online comments of otolaryngologists in the 20 most populated cities in the United States from healthgrades.com, develop and validate a natural language processing (NLP) logistic regression algorithm for automated text classification of reviews into 10 categories, and compare 1- and 5-star reviews in directly-physician-related and non-physician-related categories.

Methods

1977 1-star and 12,682 5-star reviews were collected. The primary investigator manually categorized a training dataset of 324 1-star and 909 5-star reviews, while a validation subset of 100 5-star and 50 1-star reviews underwent dual manual categorization. Using scikit-learn, an NLP algorithm was trained and validated on the subsets, with F1 scores evaluating text classification accuracy against manual categorization. The algorithm was then applied to the entire dataset with comparison of review categorization according to 1- and 5-star reviews.

Results

F1 scores for NLP validation ranged between 0.71 and 0.97. Significant associations emerged between 1-star reviews and treatment plan, accessibility, wait time, office scheduling, billing, and facilities. 5-star reviews were associated with surgery/procedure, bedside manner, and staff/mid-levels.

Conclusion

The study successfully validated an NLP text classification system for categorizing online physician reviews. Positive reviews were found to have an association with directly-physician related context. 1-star reviews were related to treatment plan, accessibility, wait time, office scheduling, billing, and facilities. This method of text classification effectively discerned the nuances of human-written text, providing valuable insights into online healthcare feedback that is scalable.

Level of evidence: Level 3

Abstract Image

查看原文本刊更多论文

耳鼻喉科医生在线评论的自动分类。

研究目的该研究旨在从healthgrades.com网站上提取美国人口最多的20个城市中耳鼻喉科医生的在线评论，开发并验证一种自然语言处理（NLP）逻辑回归算法，用于将评论文本自动分类为10个类别，并比较与医生直接相关和与医生无关类别中的1星和5星评论。方法：该研究收集了1977条1星和12682条5星评论。主要研究人员对包含 324 篇 1 星和 909 篇 5 星评论的训练数据集进行了人工分类，同时对包含 100 篇 5 星和 50 篇 1 星评论的验证子集进行了双重人工分类。使用 scikit-learn 对子集进行了 NLP 算法的训练和验证，通过 F1 分数评估文本分类的准确性与人工分类的对比。然后将该算法应用于整个数据集，并根据 1 星和 5 星评论对评论分类进行比较：结果：NLP 验证的 F1 分数介于 0.71 和 0.97 之间。一星级评论与治疗方案、可及性、等待时间、诊室安排、账单和设施之间存在显著关联。5星评价与手术/程序、床边态度和工作人员/中层相关：这项研究成功验证了用于对在线医生评论进行分类的 NLP 文本分类系统。研究发现，正面评论与与医生直接相关的内容有关。一星级评论与治疗计划、可及性、等待时间、诊室安排、账单和设施有关。这种文本分类方法有效地辨别了人类撰写文本的细微差别，为在线医疗反馈提供了有价值的见解，而且是可扩展的：证据等级：3 级。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Laryngoscope Investigative Otolaryngology OTORHINOLARYNGOLOGY-

CiteScore

3.00

自引率

0.00%

发文量

245

审稿时长

11 weeks