Varada Khanna, Nilay Bhatt, Ikgyu Shin, Mattia Rosso, Sule Tinaz, Yang Ren, Hua Xu, Vipina K Keloth
{"title":"Toward automated cognitive assessment in Parkinson's disease using pretrained language models.","authors":"Varada Khanna, Nilay Bhatt, Ikgyu Shin, Mattia Rosso, Sule Tinaz, Yang Ren, Hua Xu, Vipina K Keloth","doi":"10.1016/j.jbi.2026.105126","DOIUrl":null,"url":null,"abstract":"<p><strong>Objective: </strong>Cognitive changes in Parkinson' s Disease (PD) are often difficult to capture through formal assessments alone. First-person patient narratives offer a rich source of cognitive and emotional information, but extracting structured insights from unstructured text remains challenging due to the abstract and overlapping nature of cognitive constructs. This study aims to develop and evaluate natural language processing models to automatically identify seven cognitive categories from first-person narratives written by individuals with PD.</p><p><strong>Methods: </strong>We analyzed 299 annotated narratives from 15 individuals with PD. Three model families were compared: a Bio_ClinicalBERT-based span categorization model for nested entity recognition, a fine-tuned Meta-Llama-3-8B-Instruct model using QloRA, and GPT-4o mini evaluated under zero- and few-shot settings. In addition to narrative-level evaluation, we conducted participant-grouped cross-validation to assess performance on previously unseen participant narratives.</p><p><strong>Results: </strong>Model performance varied substantially across categories and model families. In the narrative-level evaluation, the fine-tuned Meta-Llama-3-8B-Instruct achieved the highest overall performance (lenient micro-F1: 0.796 ± 0.004; macro-F1: 0.660 ± 0.009 across five random seeds), with particularly strong performance in context-dependent categories such as thought and social interaction. Under participant-grouped cross-validation, Llama achieved a mean lenient micro-F1 of 0.787 ± 0.033 and macro-F1 of 0.639 ± 0.037, compared with 0.647 ± 0.030 and 0.425 ± 0.019 for Bio_ClinicalBERT. GPT-4o mini performed substantially lower, although five-shot prompting improved performance. Overall, this task proved more challenging than conventional information extraction due to the abstract and overlapping nature of narrative cognitive accounts.</p><p><strong>Conclusion: </strong>Fine-tuned large language models show strong potential for extracting categories related to cognitive processes from patient narratives in PD. With continued refinement, and validation in cohorts spanning the full range of cognitive status, these NLP approaches could provide a foundation for future systems designed to support longitudinal cognitive monitoring and serve as a prospective complement to formal neuropsychological assessments.</p>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":" ","pages":"105126"},"PeriodicalIF":5.9000,"publicationDate":"2026-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Biomedical Informatics","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.jbi.2026.105126","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}
引用次数: 0
Abstract
Objective: Cognitive changes in Parkinson' s Disease (PD) are often difficult to capture through formal assessments alone. First-person patient narratives offer a rich source of cognitive and emotional information, but extracting structured insights from unstructured text remains challenging due to the abstract and overlapping nature of cognitive constructs. This study aims to develop and evaluate natural language processing models to automatically identify seven cognitive categories from first-person narratives written by individuals with PD.
Methods: We analyzed 299 annotated narratives from 15 individuals with PD. Three model families were compared: a Bio_ClinicalBERT-based span categorization model for nested entity recognition, a fine-tuned Meta-Llama-3-8B-Instruct model using QloRA, and GPT-4o mini evaluated under zero- and few-shot settings. In addition to narrative-level evaluation, we conducted participant-grouped cross-validation to assess performance on previously unseen participant narratives.
Results: Model performance varied substantially across categories and model families. In the narrative-level evaluation, the fine-tuned Meta-Llama-3-8B-Instruct achieved the highest overall performance (lenient micro-F1: 0.796 ± 0.004; macro-F1: 0.660 ± 0.009 across five random seeds), with particularly strong performance in context-dependent categories such as thought and social interaction. Under participant-grouped cross-validation, Llama achieved a mean lenient micro-F1 of 0.787 ± 0.033 and macro-F1 of 0.639 ± 0.037, compared with 0.647 ± 0.030 and 0.425 ± 0.019 for Bio_ClinicalBERT. GPT-4o mini performed substantially lower, although five-shot prompting improved performance. Overall, this task proved more challenging than conventional information extraction due to the abstract and overlapping nature of narrative cognitive accounts.
Conclusion: Fine-tuned large language models show strong potential for extracting categories related to cognitive processes from patient narratives in PD. With continued refinement, and validation in cohorts spanning the full range of cognitive status, these NLP approaches could provide a foundation for future systems designed to support longitudinal cognitive monitoring and serve as a prospective complement to formal neuropsychological assessments.
期刊介绍:
The Journal of Biomedical Informatics reflects a commitment to high-quality original research papers, reviews, and commentaries in the area of biomedical informatics methodology. Although we publish articles motivated by applications in the biomedical sciences (for example, clinical medicine, health care, population health, and translational bioinformatics), the journal emphasizes reports of new methodologies and techniques that have general applicability and that form the basis for the evolving science of biomedical informatics. Articles on medical devices; evaluations of implemented systems (including clinical trials of information technologies); or papers that provide insight into a biological process, a specific disease, or treatment options would generally be more suitable for publication in other venues. Papers on applications of signal processing and image analysis are often more suitable for biomedical engineering journals or other informatics journals, although we do publish papers that emphasize the information management and knowledge representation/modeling issues that arise in the storage and use of biological signals and images. System descriptions are welcome if they illustrate and substantiate the underlying methodology that is the principal focus of the report and an effort is made to address the generalizability and/or range of application of that methodology. Note also that, given the international nature of JBI, papers that deal with specific languages other than English, or with country-specific health systems or approaches, are acceptable for JBI only if they offer generalizable lessons that are relevant to the broad JBI readership, regardless of their country, language, culture, or health system.