The effect of measurement error on the positive predictive value of PSW methods for SLD identification: How buffer zones dispel the illusion of inaccuracy
W. Joel Schneider , Dawn P. Flanagan , Christopher R. Niileksela , Joseph R. Engler
{"title":"The effect of measurement error on the positive predictive value of PSW methods for SLD identification: How buffer zones dispel the illusion of inaccuracy","authors":"W. Joel Schneider , Dawn P. Flanagan , Christopher R. Niileksela , Joseph R. Engler","doi":"10.1016/j.jsp.2023.101280","DOIUrl":null,"url":null,"abstract":"<div><p>Patterns of Strengths and Weaknesses (PSW) methods are widely used for identifying specific learning disabilities (SLDs). Several researchers, however, have reported that the diagnostic accuracy of PSW methods is unacceptably low when strict thresholds were used to identify students with SLDs. We believe these findings give a misleading impression of the magnitude of the diagnostic errors that are likely to arise in PSW assessments. In a simulation study of 10 million cases using a simplified PSW method for demonstration, most of what have been called diagnostic errors were cases in which observed scores and true scores fell on opposite sides of a strict threshold but were still within a buffer zone the size of a typical measurement error. Because small score differences do not result in meaningfully different case conceptualizations, the use of buffer zones reveals that previous estimates of the diagnostic accuracy of PSW methods are misleadingly low. We also demonstrate that diagnostic decisions become increasingly reliable when observed scores are comfortably distant from diagnostic thresholds. For practitioners, we present a flowchart and practical guidelines to improve the accuracy and stability of SLD identification decisions.</p></div>","PeriodicalId":48232,"journal":{"name":"Journal of School Psychology","volume":null,"pages":null},"PeriodicalIF":3.8000,"publicationDate":"2024-01-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of School Psychology","FirstCategoryId":"102","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0022440523001085","RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PSYCHOLOGY, SOCIAL","Score":null,"Total":0}
引用次数: 0
Abstract
Patterns of Strengths and Weaknesses (PSW) methods are widely used for identifying specific learning disabilities (SLDs). Several researchers, however, have reported that the diagnostic accuracy of PSW methods is unacceptably low when strict thresholds were used to identify students with SLDs. We believe these findings give a misleading impression of the magnitude of the diagnostic errors that are likely to arise in PSW assessments. In a simulation study of 10 million cases using a simplified PSW method for demonstration, most of what have been called diagnostic errors were cases in which observed scores and true scores fell on opposite sides of a strict threshold but were still within a buffer zone the size of a typical measurement error. Because small score differences do not result in meaningfully different case conceptualizations, the use of buffer zones reveals that previous estimates of the diagnostic accuracy of PSW methods are misleadingly low. We also demonstrate that diagnostic decisions become increasingly reliable when observed scores are comfortably distant from diagnostic thresholds. For practitioners, we present a flowchart and practical guidelines to improve the accuracy and stability of SLD identification decisions.
期刊介绍:
The Journal of School Psychology publishes original empirical articles and critical reviews of the literature on research and practices relevant to psychological and behavioral processes in school settings. JSP presents research on intervention mechanisms and approaches; schooling effects on the development of social, cognitive, mental-health, and achievement-related outcomes; assessment; and consultation. Submissions from a variety of disciplines are encouraged. All manuscripts are read by the Editor and one or more editorial consultants with the intent of providing appropriate and constructive written reviews.