Daniel R S Habib, Kiran Depala, Jack Lin, Samuel Le, Natalie McFall, Shiv S Dewan, Justin Huang, Michael W S Habib, Anthony E Bishay, Konrad Siebor, Gizem Babaoglu, Naweed I Chowdhury, Aaron C Moberly
{"title":"Patient Experiences in the Cochlear Implant Reddit Community: Comparing Human and Large Language Model Categorization.","authors":"Daniel R S Habib, Kiran Depala, Jack Lin, Samuel Le, Natalie McFall, Shiv S Dewan, Justin Huang, Michael W S Habib, Anthony E Bishay, Konrad Siebor, Gizem Babaoglu, Naweed I Chowdhury, Aaron C Moberly","doi":"10.1044/2025_AJA-25-00216","DOIUrl":null,"url":null,"abstract":"<p><strong>Purpose: </strong>Although some work has leveraged automated analyses of online communities to gain cochlear implant (CI) patient insights, there remains a gap in comparing human versus automated analysis of the nuanced, real-world experiences patients share outside clinical settings. This study characterizes experiences within the r/Cochlearimplants Reddit community and compares human to large language model (LLM) performance in annotating posts.</p><p><strong>Method: </strong>Using reflexive thematic analysis, 996 publicly available r/Cochlearimplants posts (October 2024-June 2025) were manually coded and consolidated into themes. Three LLMs-OpenAI o3, Gemini 2.5 Pro, and Claude Sonnet 4-were prompted with the posts and human-generated codebook to perform post categorization. Model performance was evaluated against human coding using Cohen's kappa, percent agreement, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and time.</p><p><strong>Results: </strong>Five themes emerged. Community engagement and support (<i>N</i> = 944, 94.8%) frequently involved eliciting advice (<i>N</i> = 721, 72.4%), seeking shared experiences (<i>N</i> = 249, 25.0%), and sharing negative experiences (<i>N</i> = 247, 24.8%). Other themes included the medical/surgical journey (<i>N</i> = 463, 46.5%), device/technical issues (<i>N</i> = 343, 34.4%), daily life/adjustments (<i>N</i> = 236, 23.7%), and media/outreach (7.2%, <i>N</i> = 72). OpenAI o3 and Gemini 2.5 Pro achieved the highest interrater reliability with human annotators (κ = .35 and κ = .34, respectively). OpenAI o3 had higher sensitivity (46.7%) but lower specificity (90.4%) than Gemini 2.5 Pro, which had the highest specificity (93.4%) but lower sensitivity (38.0%). Claude Sonnet 4 showed the lowest agreement (κ = .25) and PPV (30.9%). Compared to human annotation requiring 52 hr across all annotators, each LLM required less than 20 min.</p><p><strong>Conclusions: </strong>Reddit posts revealed rich discourse across CI topics. LLMs demonstrated fair agreement with human coders and can quickly aid in large-scale qualitative analysis. Although careful model selection and human expertise remain essential for accurate interpretation, LLM annotation shows potential for real-time monitoring of patient concerns to inform counseling, rehabilitation strategies, and iterative device design.</p><p><strong>Supplemental material: </strong>https://doi.org/10.23641/asha.31362847.</p>","PeriodicalId":49241,"journal":{"name":"American Journal of Audiology","volume":" ","pages":"487-496"},"PeriodicalIF":1.7000,"publicationDate":"2026-06-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"American Journal of Audiology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1044/2025_AJA-25-00216","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2026/2/26 0:00:00","PubModel":"Epub","JCR":"Q3","JCRName":"AUDIOLOGY & SPEECH-LANGUAGE PATHOLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Purpose: Although some work has leveraged automated analyses of online communities to gain cochlear implant (CI) patient insights, there remains a gap in comparing human versus automated analysis of the nuanced, real-world experiences patients share outside clinical settings. This study characterizes experiences within the r/Cochlearimplants Reddit community and compares human to large language model (LLM) performance in annotating posts.
Method: Using reflexive thematic analysis, 996 publicly available r/Cochlearimplants posts (October 2024-June 2025) were manually coded and consolidated into themes. Three LLMs-OpenAI o3, Gemini 2.5 Pro, and Claude Sonnet 4-were prompted with the posts and human-generated codebook to perform post categorization. Model performance was evaluated against human coding using Cohen's kappa, percent agreement, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and time.
Results: Five themes emerged. Community engagement and support (N = 944, 94.8%) frequently involved eliciting advice (N = 721, 72.4%), seeking shared experiences (N = 249, 25.0%), and sharing negative experiences (N = 247, 24.8%). Other themes included the medical/surgical journey (N = 463, 46.5%), device/technical issues (N = 343, 34.4%), daily life/adjustments (N = 236, 23.7%), and media/outreach (7.2%, N = 72). OpenAI o3 and Gemini 2.5 Pro achieved the highest interrater reliability with human annotators (κ = .35 and κ = .34, respectively). OpenAI o3 had higher sensitivity (46.7%) but lower specificity (90.4%) than Gemini 2.5 Pro, which had the highest specificity (93.4%) but lower sensitivity (38.0%). Claude Sonnet 4 showed the lowest agreement (κ = .25) and PPV (30.9%). Compared to human annotation requiring 52 hr across all annotators, each LLM required less than 20 min.
Conclusions: Reddit posts revealed rich discourse across CI topics. LLMs demonstrated fair agreement with human coders and can quickly aid in large-scale qualitative analysis. Although careful model selection and human expertise remain essential for accurate interpretation, LLM annotation shows potential for real-time monitoring of patient concerns to inform counseling, rehabilitation strategies, and iterative device design.
期刊介绍：
Mission: AJA publishes peer-reviewed research and other scholarly articles pertaining to clinical audiology methods and issues, and serves as an outlet for discussion of related professional and educational issues and ideas. The journal is an international outlet for research on clinical research pertaining to screening, diagnosis, management and outcomes of hearing and balance disorders as well as the etiologies and characteristics of these disorders. The clinical orientation of the journal allows for the publication of reports on audiology as implemented nationally and internationally, including novel clinical procedures, approaches, and cases. AJA seeks to advance evidence-based practice by disseminating the results of new studies as well as providing a forum for critical reviews and meta-analyses of previously published work.
Scope: The broad field of clinical audiology, including audiologic/aural rehabilitation; balance and balance disorders; cultural and linguistic diversity; detection, diagnosis, prevention, habilitation, rehabilitation, and monitoring of hearing loss; hearing aids, cochlear implants, and hearing-assistive technology; hearing disorders; lifespan perspectives on auditory function; speech perception; and tinnitus.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
