Developing Validity Evidence for a Novel Pediatric Telemedicine Assessment Tool (PTAT) for Pediatric Residents: A Pilot Study.

Abstract

Objectives: Telemedicine use among pediatric residents has grown in recent years, accompanied by the development of best practices and curricula that outline essential telemedicine skills. However, there is a notable lack of methods to assess these skills. Our work aims to address this gap by designing and building validity evidence for a novel pediatric telemedicine assessment tool, the Pediatric Telemedicine Assessment Tool (PTAT).

Methods: The PTAT was created by the study team. Multiple sources of validity evidence were collected, including content validity, internal structure, response process, and relationship to other variables. The PTAT was studied using standardized patient telemedicine encounters with pediatric residents. Twenty-one residents performed a telemedicine encounter that was scored using the PTAT. Recruited faculty scored the encounters and were surveyed on PTAT usability using a modified Systems Usability Scale (SUS). Descriptive statistics and interrater reliability (intraclass correlation (ICC)) were calculated.

Results: The PTAT consists of 16 items scored on a four-point rating scale; it also includes a five-point overall global skills assessment (GSA). Compared to post-graduate year-1 (PGY-1) residents, PGY-2-4 residents had more telemedicine experience (92% vs. 25%, p=0.003) and higher PTAT (1.39 vs 1.21, p=0.02) and GSA scores (1.67 vs. 1.00, p=0.002). The PTAT items had moderate overall interrater reliability (ICC=0.6 (95% CI=0.50-0.68)). The faculty reported "above average" usability.

Conclusions: The PTAT demonstrated multiple sources of validity evidence including content validity, internal structure, response process, and relationship to other variables. It may be used as a tool to structure formative feedback for resident use of telemedicine in standardized patient encounters.