Development and Performance Validation of an Automated Generative Pretrained Transformer–Based Evaluation Tool for the Physiotherapy Evidence Database Scale

Abstract

Objectives

To develop an automated generative pretrained transformer (GPT)- physiotherapy evidence database (PEDro) evaluation tool (GPT-PEDro ET) and validate its performance (including intrarater reliability and concurrent validity) for evaluating the methodological quality of RCTs using the PEDro scale.

Design

A psychometric validation study with a repeated measurements design.

Setting

Research laboratory.

Participants

One hundred and twenty-five RCTs on neurofacilitation interventions in stroke rehabilitation were retrieved from the PEDro database.

Interventions

Not applicable.

Main Outcomes and Measures

The primary outcome was RCT quality scores evaluated using the GPT-PEDro ET tool at both total score and individual item levels across 2 rounds of evaluation. Intrarater reliability was determined using the intraclass correlation coefficient (ICC) for total score and prevalence-adjusted and bias-adjusted κ (PABAK) for individual items. Concurrent validity was assessed with ICC (total score) and PABAK (individual item), Bland–Altman analysis with 95% limits of agreement, and heteroscedasticity testing.

Results

The GPT-PEDro ET tool achieved almost perfect intrarater reliability (total score ICC=1.00; individual item PABAK=0.94-1.00). The concurrent validity was moderate to high at both the total score level (ICCs=0.83, 0.83) and the individual item level for all the items (PABAKs=0.68-0.97).

Conclusions

The results suggest that the GPT-PEDro ET tool may be a useful automated tool for evaluating the methodological quality of RCTs. It shows potential for reducing manual evaluation workload and supporting clinical and research applications.