Toward automated cognitive assessment in Parkinson's disease using pretrained language models.

IF 5.9 2区 医学 Q2 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS
Varada Khanna, Nilay Bhatt, Ikgyu Shin, Mattia Rosso, Sule Tinaz, Yang Ren, Hua Xu, Vipina K Keloth
{"title":"Toward automated cognitive assessment in Parkinson's disease using pretrained language models.","authors":"Varada Khanna, Nilay Bhatt, Ikgyu Shin, Mattia Rosso, Sule Tinaz, Yang Ren, Hua Xu, Vipina K Keloth","doi":"10.1016/j.jbi.2026.105126","DOIUrl":null,"url":null,"abstract":"<p><strong>Objective: </strong>Cognitive changes in Parkinson' s Disease (PD) are often difficult to capture through formal assessments alone. First-person patient narratives offer a rich source of cognitive and emotional information, but extracting structured insights from unstructured text remains challenging due to the abstract and overlapping nature of cognitive constructs. This study aims to develop and evaluate natural language processing models to automatically identify seven cognitive categories from first-person narratives written by individuals with PD.</p><p><strong>Methods: </strong>We analyzed 299 annotated narratives from 15 individuals with PD. Three model families were compared: a Bio_ClinicalBERT-based span categorization model for nested entity recognition, a fine-tuned Meta-Llama-3-8B-Instruct model using QloRA, and GPT-4o mini evaluated under zero- and few-shot settings. In addition to narrative-level evaluation, we conducted participant-grouped cross-validation to assess performance on previously unseen participant narratives.</p><p><strong>Results: </strong>Model performance varied substantially across categories and model families. In the narrative-level evaluation, the fine-tuned Meta-Llama-3-8B-Instruct achieved the highest overall performance (lenient micro-F1: 0.796 ± 0.004; macro-F1: 0.660 ± 0.009 across five random seeds), with particularly strong performance in context-dependent categories such as thought and social interaction. Under participant-grouped cross-validation, Llama achieved a mean lenient micro-F1 of 0.787 ± 0.033 and macro-F1 of 0.639 ± 0.037, compared with 0.647 ± 0.030 and 0.425 ± 0.019 for Bio_ClinicalBERT. GPT-4o mini performed substantially lower, although five-shot prompting improved performance. Overall, this task proved more challenging than conventional information extraction due to the abstract and overlapping nature of narrative cognitive accounts.</p><p><strong>Conclusion: </strong>Fine-tuned large language models show strong potential for extracting categories related to cognitive processes from patient narratives in PD. With continued refinement, and validation in cohorts spanning the full range of cognitive status, these NLP approaches could provide a foundation for future systems designed to support longitudinal cognitive monitoring and serve as a prospective complement to formal neuropsychological assessments.</p>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":" ","pages":"105126"},"PeriodicalIF":5.9000,"publicationDate":"2026-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Biomedical Informatics","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.jbi.2026.105126","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}
引用次数: 0

Abstract

Objective: Cognitive changes in Parkinson' s Disease (PD) are often difficult to capture through formal assessments alone. First-person patient narratives offer a rich source of cognitive and emotional information, but extracting structured insights from unstructured text remains challenging due to the abstract and overlapping nature of cognitive constructs. This study aims to develop and evaluate natural language processing models to automatically identify seven cognitive categories from first-person narratives written by individuals with PD.

Methods: We analyzed 299 annotated narratives from 15 individuals with PD. Three model families were compared: a Bio_ClinicalBERT-based span categorization model for nested entity recognition, a fine-tuned Meta-Llama-3-8B-Instruct model using QloRA, and GPT-4o mini evaluated under zero- and few-shot settings. In addition to narrative-level evaluation, we conducted participant-grouped cross-validation to assess performance on previously unseen participant narratives.

Results: Model performance varied substantially across categories and model families. In the narrative-level evaluation, the fine-tuned Meta-Llama-3-8B-Instruct achieved the highest overall performance (lenient micro-F1: 0.796 ± 0.004; macro-F1: 0.660 ± 0.009 across five random seeds), with particularly strong performance in context-dependent categories such as thought and social interaction. Under participant-grouped cross-validation, Llama achieved a mean lenient micro-F1 of 0.787 ± 0.033 and macro-F1 of 0.639 ± 0.037, compared with 0.647 ± 0.030 and 0.425 ± 0.019 for Bio_ClinicalBERT. GPT-4o mini performed substantially lower, although five-shot prompting improved performance. Overall, this task proved more challenging than conventional information extraction due to the abstract and overlapping nature of narrative cognitive accounts.

Conclusion: Fine-tuned large language models show strong potential for extracting categories related to cognitive processes from patient narratives in PD. With continued refinement, and validation in cohorts spanning the full range of cognitive status, these NLP approaches could provide a foundation for future systems designed to support longitudinal cognitive monitoring and serve as a prospective complement to formal neuropsychological assessments.

使用预训练语言模型对帕金森病进行自动认知评估。
目的:帕金森病(PD)的认知变化通常很难通过单独的正式评估来捕捉。第一人称患者叙述提供了丰富的认知和情感信息来源,但由于认知结构的抽象性和重叠性,从非结构化文本中提取结构化的见解仍然具有挑战性。本研究旨在开发和评估自然语言处理模型,从PD患者的第一人称叙述中自动识别7种认知类别。方法:对15例PD患者299份注释叙述进行分析。比较了三种模型:基于bio_clinicalbert的嵌套实体识别跨分类模型,使用QloRA的微调meta - lama-3- 8b - instruct模型,以及在零次和少次射击设置下评估的gpt - 40 mini模型。除了叙事层面的评估,我们还进行了参与者分组交叉验证,以评估先前未见过的参与者叙事的表现。结果:模型性能在不同类别和模型家族之间有很大差异。在叙事层面的评估中,经过微调的Meta-Llama-3-8B-Instruct在五个随机种子中获得了最高的整体表现(宽松的微观f1: 0.796 ± 0.004;宏观f1: 0.660 ± 0.009),在思维和社会互动等情境依赖类别中表现尤为突出。在参与者分组交叉验证中,Llama的微观f1均值为0.787 ± 0.033,宏观f1均值为0.639 ± 0.037,而Bio_ClinicalBERT的微观f1均值为0.647 ± 0.030,宏观f1均值为0.425 ± 0.019。gpt - 40 mini的表现要低得多,尽管5发助推弹提高了性能。总的来说,由于叙述性认知账户的抽象性和重叠性,这项任务比传统的信息提取更具挑战性。结论:微调的大型语言模型显示出从PD患者叙述中提取与认知过程相关的类别的强大潜力。随着不断的改进,并在跨越认知状态的所有队列中进行验证,这些NLP方法可以为未来设计支持纵向认知监测的系统提供基础,并作为正式神经心理学评估的前瞻性补充。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Journal of Biomedical Informatics
Journal of Biomedical Informatics 医学-计算机:跨学科应用
CiteScore
8.90
自引率
6.70%
发文量
243
审稿时长
32 days
期刊介绍: The Journal of Biomedical Informatics reflects a commitment to high-quality original research papers, reviews, and commentaries in the area of biomedical informatics methodology. Although we publish articles motivated by applications in the biomedical sciences (for example, clinical medicine, health care, population health, and translational bioinformatics), the journal emphasizes reports of new methodologies and techniques that have general applicability and that form the basis for the evolving science of biomedical informatics. Articles on medical devices; evaluations of implemented systems (including clinical trials of information technologies); or papers that provide insight into a biological process, a specific disease, or treatment options would generally be more suitable for publication in other venues. Papers on applications of signal processing and image analysis are often more suitable for biomedical engineering journals or other informatics journals, although we do publish papers that emphasize the information management and knowledge representation/modeling issues that arise in the storage and use of biological signals and images. System descriptions are welcome if they illustrate and substantiate the underlying methodology that is the principal focus of the report and an effort is made to address the generalizability and/or range of application of that methodology. Note also that, given the international nature of JBI, papers that deal with specific languages other than English, or with country-specific health systems or approaches, are acceptable for JBI only if they offer generalizable lessons that are relevant to the broad JBI readership, regardless of their country, language, culture, or health system.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书