Theresa E Scott, Anjile An, Nicole Paradise Black, Su-Ting T Li, Erika L Abramson
{"title":"为儿科住院医师开发新型儿科远程医疗评估工具(PTAT)的效度证据:一项试点研究。","authors":"Theresa E Scott, Anjile An, Nicole Paradise Black, Su-Ting T Li, Erika L Abramson","doi":"10.1016/j.acap.2025.103145","DOIUrl":null,"url":null,"abstract":"<p><strong>Objectives: </strong>Telemedicine use among pediatric residents has grown in recent years, accompanied by the development of best practices and curricula that outline essential telemedicine skills. However, there is a notable lack of methods to assess these skills. Our work aims to address this gap by designing and building validity evidence for a novel pediatric telemedicine assessment tool, the Pediatric Telemedicine Assessment Tool (PTAT).</p><p><strong>Methods: </strong>The PTAT was created by the study team. Multiple sources of validity evidence were collected, including content validity, internal structure, response process, and relationship to other variables. The PTAT was studied using standardized patient telemedicine encounters with pediatric residents. Twenty-one residents performed a telemedicine encounter that was scored using the PTAT. Recruited faculty scored the encounters and were surveyed on PTAT usability using a modified Systems Usability Scale (SUS). Descriptive statistics and interrater reliability (intraclass correlation (ICC)) were calculated.</p><p><strong>Results: </strong>The PTAT consists of 16 items scored on a four-point rating scale; it also includes a five-point overall global skills assessment (GSA). Compared to post-graduate year-1 (PGY-1) residents, PGY-2-4 residents had more telemedicine experience (92% vs. 25%, p=0.003) and higher PTAT (1.39 vs 1.21, p=0.02) and GSA scores (1.67 vs. 1.00, p=0.002). The PTAT items had moderate overall interrater reliability (ICC=0.6 (95% CI=0.50-0.68)). The faculty reported \"above average\" usability.</p><p><strong>Conclusions: </strong>The PTAT demonstrated multiple sources of validity evidence including content validity, internal structure, response process, and relationship to other variables. It may be used as a tool to structure formative feedback for resident use of telemedicine in standardized patient encounters.</p>","PeriodicalId":50930,"journal":{"name":"Academic Pediatrics","volume":" ","pages":"103145"},"PeriodicalIF":2.8000,"publicationDate":"2025-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Developing Validity Evidence for a Novel Pediatric Telemedicine Assessment Tool (PTAT) for Pediatric Residents: A Pilot Study.\",\"authors\":\"Theresa E Scott, Anjile An, Nicole Paradise Black, Su-Ting T Li, Erika L Abramson\",\"doi\":\"10.1016/j.acap.2025.103145\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><strong>Objectives: </strong>Telemedicine use among pediatric residents has grown in recent years, accompanied by the development of best practices and curricula that outline essential telemedicine skills. However, there is a notable lack of methods to assess these skills. Our work aims to address this gap by designing and building validity evidence for a novel pediatric telemedicine assessment tool, the Pediatric Telemedicine Assessment Tool (PTAT).</p><p><strong>Methods: </strong>The PTAT was created by the study team. Multiple sources of validity evidence were collected, including content validity, internal structure, response process, and relationship to other variables. The PTAT was studied using standardized patient telemedicine encounters with pediatric residents. Twenty-one residents performed a telemedicine encounter that was scored using the PTAT. Recruited faculty scored the encounters and were surveyed on PTAT usability using a modified Systems Usability Scale (SUS). Descriptive statistics and interrater reliability (intraclass correlation (ICC)) were calculated.</p><p><strong>Results: </strong>The PTAT consists of 16 items scored on a four-point rating scale; it also includes a five-point overall global skills assessment (GSA). Compared to post-graduate year-1 (PGY-1) residents, PGY-2-4 residents had more telemedicine experience (92% vs. 25%, p=0.003) and higher PTAT (1.39 vs 1.21, p=0.02) and GSA scores (1.67 vs. 1.00, p=0.002). The PTAT items had moderate overall interrater reliability (ICC=0.6 (95% CI=0.50-0.68)). The faculty reported \\\"above average\\\" usability.</p><p><strong>Conclusions: </strong>The PTAT demonstrated multiple sources of validity evidence including content validity, internal structure, response process, and relationship to other variables. It may be used as a tool to structure formative feedback for resident use of telemedicine in standardized patient encounters.</p>\",\"PeriodicalId\":50930,\"journal\":{\"name\":\"Academic Pediatrics\",\"volume\":\" \",\"pages\":\"103145\"},\"PeriodicalIF\":2.8000,\"publicationDate\":\"2025-09-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Academic Pediatrics\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://doi.org/10.1016/j.acap.2025.103145\",\"RegionNum\":3,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"PEDIATRICS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Academic Pediatrics","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.acap.2025.103145","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PEDIATRICS","Score":null,"Total":0}
Developing Validity Evidence for a Novel Pediatric Telemedicine Assessment Tool (PTAT) for Pediatric Residents: A Pilot Study.
Objectives: Telemedicine use among pediatric residents has grown in recent years, accompanied by the development of best practices and curricula that outline essential telemedicine skills. However, there is a notable lack of methods to assess these skills. Our work aims to address this gap by designing and building validity evidence for a novel pediatric telemedicine assessment tool, the Pediatric Telemedicine Assessment Tool (PTAT).
Methods: The PTAT was created by the study team. Multiple sources of validity evidence were collected, including content validity, internal structure, response process, and relationship to other variables. The PTAT was studied using standardized patient telemedicine encounters with pediatric residents. Twenty-one residents performed a telemedicine encounter that was scored using the PTAT. Recruited faculty scored the encounters and were surveyed on PTAT usability using a modified Systems Usability Scale (SUS). Descriptive statistics and interrater reliability (intraclass correlation (ICC)) were calculated.
Results: The PTAT consists of 16 items scored on a four-point rating scale; it also includes a five-point overall global skills assessment (GSA). Compared to post-graduate year-1 (PGY-1) residents, PGY-2-4 residents had more telemedicine experience (92% vs. 25%, p=0.003) and higher PTAT (1.39 vs 1.21, p=0.02) and GSA scores (1.67 vs. 1.00, p=0.002). The PTAT items had moderate overall interrater reliability (ICC=0.6 (95% CI=0.50-0.68)). The faculty reported "above average" usability.
Conclusions: The PTAT demonstrated multiple sources of validity evidence including content validity, internal structure, response process, and relationship to other variables. It may be used as a tool to structure formative feedback for resident use of telemedicine in standardized patient encounters.
期刊介绍:
Academic Pediatrics, the official journal of the Academic Pediatric Association, is a peer-reviewed publication whose purpose is to strengthen the research and educational base of academic general pediatrics. The journal provides leadership in pediatric education, research, patient care and advocacy. Content areas include pediatric education, emergency medicine, injury, abuse, behavioral pediatrics, holistic medicine, child health services and health policy,and the environment. The journal provides an active forum for the presentation of pediatric educational research in diverse settings, involving medical students, residents, fellows, and practicing professionals. The journal also emphasizes important research relating to the quality of child health care, health care policy, and the organization of child health services. It also includes systematic reviews of primary care interventions and important methodologic papers to aid research in child health and education.