Content validity of AI-generated stuttering assessment and intervention programs based on expert review: A comparative analysis across age groups and language versions

IF 2.8 3区医学 Q1 AUDIOLOGY & SPEECH-LANGUAGE PATHOLOGY

Journal of Fluency Disorders Pub Date : 2026-03-01 Epub Date: 2025-12-03 DOI:10.1016/j.jfludis.2025.106186

Ayşe Nur Koçak , Melis Buse Arslan

{"title":"Content validity of AI-generated stuttering assessment and intervention programs based on expert review: A comparative analysis across age groups and language versions","authors":"Ayşe Nur Koçak , Melis Buse Arslan","doi":"10.1016/j.jfludis.2025.106186","DOIUrl":null,"url":null,"abstract":"<div><h3>Purpose</h3><div>This study aimed to evaluate the content validity and inter-rater reliability of stuttering assessment and intervention programs generated by artificial intelligence (GPT-4) in both Turkish and English for preschool, school-age, and adult populations. It also examined whether linguistic or cultural differences affected expert evaluations.</div></div><div><h3>Methods</h3><div>Twelve AI-generated programs (six in Turkish, six in English) were reviewed by twelve certified speech-language pathologists specializing in fluency disorders. Each item was rated using a 5-point Likert scale. Descriptive statistics, Cronbach’s Alpha, and Intraclass Correlation Coefficients (ICC) were calculated to assess consistency and reliability.</div></div><div><h3>Results</h3><div>The majority of items were rated as appropriate or highly appropriate (M = 4.6–4.9). The overall reliability among raters was poor (ICC = 0.45), while single-rater reliability was higher (ICC = 0.65). Only a small number of items were flagged for revision, typically involving emotional or contextual components. Experts noted that English versions tended to be more detailed and literature-consistent, whereas certain Turkish terms required clearer cultural adaptation.</div></div><div><h3>Conclusion</h3><div>GPT-4 can produce clinically relevant and linguistically accurate stuttering materials when paired with expert review. However, human validation remains essential to refine affective and culture-specific elements. These findings support the integration of AI-assisted tools in multilingual clinical content development.</div></div>","PeriodicalId":49166,"journal":{"name":"Journal of Fluency Disorders","volume":"87 ","pages":"Article 106186"},"PeriodicalIF":2.8000,"publicationDate":"2026-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Fluency Disorders","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0094730X25000889","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/12/3 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"AUDIOLOGY & SPEECH-LANGUAGE PATHOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Purpose

This study aimed to evaluate the content validity and inter-rater reliability of stuttering assessment and intervention programs generated by artificial intelligence (GPT-4) in both Turkish and English for preschool, school-age, and adult populations. It also examined whether linguistic or cultural differences affected expert evaluations.

Methods

Twelve AI-generated programs (six in Turkish, six in English) were reviewed by twelve certified speech-language pathologists specializing in fluency disorders. Each item was rated using a 5-point Likert scale. Descriptive statistics, Cronbach’s Alpha, and Intraclass Correlation Coefficients (ICC) were calculated to assess consistency and reliability.

Results

The majority of items were rated as appropriate or highly appropriate (M = 4.6–4.9). The overall reliability among raters was poor (ICC = 0.45), while single-rater reliability was higher (ICC = 0.65). Only a small number of items were flagged for revision, typically involving emotional or contextual components. Experts noted that English versions tended to be more detailed and literature-consistent, whereas certain Turkish terms required clearer cultural adaptation.

Conclusion

GPT-4 can produce clinically relevant and linguistically accurate stuttering materials when paired with expert review. However, human validation remains essential to refine affective and culture-specific elements. These findings support the integration of AI-assisted tools in multilingual clinical content development.

查看原文本刊更多论文

基于专家评审的人工智能生成的口吃评估和干预方案的内容效度：跨年龄组和语言版本的比较分析

目的本研究旨在评估人工智能生成的口吃评估和干预方案（GPT-4）在土耳其语和英语中的内容效度和量表间信度，该方案适用于学龄前、学龄和成年人群。它还研究了语言或文化差异是否会影响专家的评估。方法12个人工智能生成的程序（6个土耳其语，6个英语）由12位专业从事流利性障碍的认证语言病理学家进行审查。每个项目都使用5分李克特量表进行评分。计算描述性统计、Cronbach’s Alpha和类内相关系数（ICC）来评估一致性和可靠性。结果大多数项目被评为适宜或高度适宜（M = 4.6 ~ 4.9）。评分者的整体信度较差（ICC = 0.45），而单评分者的信度较高（ICC = 0.65）。只有少数项目被标记为需要修改，通常涉及情感或上下文成分。专家指出，英文版本往往更详细和文献一致，而某些土耳其语术语需要更明确的文化适应。结论pt -4配合专家评语，可获得临床相关、语言准确的口吃材料。然而，人类验证对于完善情感和文化特定元素仍然至关重要。这些发现支持将人工智能辅助工具整合到多语言临床内容开发中。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Fluency Disorders AUDIOLOGY & SPEECH-LANGUAGE PATHOLOGY-REHABILITATION

CiteScore

3.70

自引率

14.30%

发文量

审稿时长

>12 weeks

期刊介绍： Journal of Fluency Disorders provides comprehensive coverage of clinical, experimental, and theoretical aspects of stuttering, including the latest remediation techniques. As the official journal of the International Fluency Association, the journal features full-length research and clinical reports; methodological, theoretical and philosophical articles; reviews; short communications and much more – all readily accessible and tailored to the needs of the professional.