Cognitive status assessment of older adults - test administration by conversational artificial intelligence (AI) chatbot: proof-of-concept investigation.

IF 1.7 4区心理学 Q3 CLINICAL NEUROLOGY

Journal of clinical and experimental neuropsychology Pub Date : 2025-07-01 Epub Date: 2025-08-12 DOI:10.1080/13803395.2025.2542248

Anastasia Serafimovska, Katrina Swavley, Alice Zhang Qian Ao, Kirsten L Challinor, Tony Florio

{"title":"Cognitive status assessment of older adults - test administration by conversational artificial intelligence (AI) chatbot: proof-of-concept investigation.","authors":"Anastasia Serafimovska, Katrina Swavley, Alice Zhang Qian Ao, Kirsten L Challinor, Tony Florio","doi":"10.1080/13803395.2025.2542248","DOIUrl":null,"url":null,"abstract":"Background: The Telephone Interview for Cognitive Status-Modified (TICS-M) is a widely utilized tool for remotely assessing cognitive function, particularly among community-dwelling older adults who are unable to attend in-person evaluations. In healthcare, AI has the potential to enhance service delivery by increasing efficiency, expanding accessibility, and reducing the cost per service. Using a conversational AI chatbot, we automated administration of TICS-M (traditionally administered by psychologists), referring to this chatbot-administered version as TICS-M-AI. The aim was to investigate proof-of-concept for chatbot automation of cognitive assessment. We report three studies evaluating psychometric properties of TICS-M-AI and an additional study on safety.Method: Study1: Concurrent validity of the TICS-M-AI was assessed by administration of the TICS-M (by Psychologist) and the TICS-M-AI to the same participants (n = 100), one week apart. Study 2: Test-retest reliability was assessed by administering the TICS-M-AI twice to each participant, one week apart (n = 82) and comparing results. Study 3: Construct validity was assessed by attempted replication, using TICS-M-AI data (n = 264), of a previously published study by Lindgren et al. (2019) of item response patterns observed using data obtained by traditional clinician administered TICS-M. Study 4: Safety was assessed by comparing rates of reported assessment-related distress between TICS-M (n = 100) and TICS-M-AI (n = 264) administrations.Results: TICS-M-AI concurrent validity (r = 0.81, 88% classification agreement, κ = 0.73) with the TICS-M and good test-retest reliability (r = 0.76, ICC = 0.72, 83% agreement, κ = 0.65). Using the TICS-M-AI we replicated Lindgren et al. (2019) result which used the TICS-M.Conclusions: TICS-M-AI administered by an AI chatbot performed well compared to traditional TICS-M administration by a psychologist. TICS-M-AI is reliable, valid, and equally safe with added advantages of lower cost, scalability, and broader accessibility. Future research should address generalizability across diverse populations and refine AI adaptability.","PeriodicalId":15382,"journal":{"name":"Journal of clinical and experimental neuropsychology","volume":" ","pages":"472-484"},"PeriodicalIF":1.7000,"publicationDate":"2025-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of clinical and experimental neuropsychology","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1080/13803395.2025.2542248","RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/8/12 0:00:00","PubModel":"Epub","JCR":"Q3","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Background: The Telephone Interview for Cognitive Status-Modified (TICS-M) is a widely utilized tool for remotely assessing cognitive function, particularly among community-dwelling older adults who are unable to attend in-person evaluations. In healthcare, AI has the potential to enhance service delivery by increasing efficiency, expanding accessibility, and reducing the cost per service. Using a conversational AI chatbot, we automated administration of TICS-M (traditionally administered by psychologists), referring to this chatbot-administered version as TICS-M-AI. The aim was to investigate proof-of-concept for chatbot automation of cognitive assessment. We report three studies evaluating psychometric properties of TICS-M-AI and an additional study on safety.

Method: Study1: Concurrent validity of the TICS-M-AI was assessed by administration of the TICS-M (by Psychologist) and the TICS-M-AI to the same participants (n = 100), one week apart. Study 2: Test-retest reliability was assessed by administering the TICS-M-AI twice to each participant, one week apart (n = 82) and comparing results. Study 3: Construct validity was assessed by attempted replication, using TICS-M-AI data (n = 264), of a previously published study by Lindgren et al. (2019) of item response patterns observed using data obtained by traditional clinician administered TICS-M. Study 4: Safety was assessed by comparing rates of reported assessment-related distress between TICS-M (n = 100) and TICS-M-AI (n = 264) administrations.

Results: TICS-M-AI concurrent validity (r = 0.81, 88% classification agreement, κ = 0.73) with the TICS-M and good test-retest reliability (r = 0.76, ICC = 0.72, 83% agreement, κ = 0.65). Using the TICS-M-AI we replicated Lindgren et al. (2019) result which used the TICS-M.

Conclusions: TICS-M-AI administered by an AI chatbot performed well compared to traditional TICS-M administration by a psychologist. TICS-M-AI is reliable, valid, and equally safe with added advantages of lower cost, scalability, and broader accessibility. Future research should address generalizability across diverse populations and refine AI adaptability.

查看原文本刊更多论文

老年人的认知状态评估-会话人工智能（AI）聊天机器人的测试管理：概念验证调查。

背景：认知状态修正电话访谈（tic - m）是一种广泛使用的远程评估认知功能的工具，特别是在无法参加现场评估的社区老年人中。在医疗保健领域，人工智能有潜力通过提高效率、扩大可及性和降低每项服务的成本来改善服务提供。使用会话AI聊天机器人，我们自动管理tic - m（传统上由心理学家管理），将这种聊天机器人管理的版本称为tic - m -AI。目的是研究认知评估的聊天机器人自动化的概念验证。我们报告了三项评估tic - m - ai心理测量特性的研究和一项关于安全性的研究。方法：研究1：通过对同一参与者（n = 100）分别使用tic - m（由心理学家）和tic - m - ai，间隔一周，评估tic - m - ai的同时效度。研究2：通过对每位参与者进行两次TICS-M-AI，间隔一周（n = 82）并比较结果来评估重测信度。研究3：通过使用TICS-M- ai数据（n = 264），通过尝试复制Lindgren等人（2019）先前发表的研究来评估结构效度，该研究使用传统临床医生管理的TICS-M获得的数据观察到项目反应模式。研究4：通过比较TICS-M （n = 100）和TICS-M- ai （n = 264）两组报告的评估相关窘迫率来评估安全性。结果：TICS-M- ai的并发效度（r = 0.81, 88%分类一致性，κ = 0.73）与TICS-M具有良好的重测信度（r = 0.76, ICC = 0.72, 83%一致性，κ = 0.65）。使用TICS-M- ai，我们复制了Lindgren等人（2019）使用TICS-M的结果。结论：与传统的由心理学家管理的tic - m相比，由AI聊天机器人管理的tic - m -AI表现良好。tic - m - ai可靠、有效且同样安全，还具有成本更低、可扩展性和更广泛的可访问性等额外优势。未来的研究应该解决不同人群的普遍性，并完善人工智能的适应性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of clinical and experimental neuropsychology 医学-临床神经学

CiteScore

3.20

自引率

4.50%

发文量

审稿时长

6-12 weeks

期刊介绍： Journal of Clinical and Experimental Neuropsychology ( JCEN) publishes research on the neuropsychological consequences of brain disease, disorders, and dysfunction, and aims to promote the integration of theories, methods, and research findings in clinical and experimental neuropsychology. The primary emphasis of JCEN is to publish original empirical research pertaining to brain-behavior relationships and neuropsychological manifestations of brain disease. Theoretical and methodological papers, critical reviews of content areas, and theoretically-relevant case studies are also welcome.