Reliability Assessment of an Observation Tool to Measure Praise Characteristics

IF 1.4 3区心理学 Q3 EDUCATION, SPECIAL

Journal of Positive Behavior Interventions Pub Date : 2021-01-01 DOI:10.1177/1098300720907988

Andrew M. Markelz, Benjamin S. Riden, Kimberly A. Zoder-Martell, Joseph E. Miller, Sarah J. Bolinger

{"title":"Reliability Assessment of an Observation Tool to Measure Praise Characteristics","authors":"Andrew M. Markelz, Benjamin S. Riden, Kimberly A. Zoder-Martell, Joseph E. Miller, Sarah J. Bolinger","doi":"10.1177/1098300720907988","DOIUrl":null,"url":null,"abstract":"Supported by decades of research on praise and its effect on student behaviors, we developed the Behavior-Specific Praise–Observation Tool (BSP-OT) to measure characteristics of effective praise. We evaluated interrater reliability of the BSP-OT to measure praise specificity, contingency, and variety using intraclass correlation (ICC) and Cohen’s kappa statistics. In addition, we assessed usefulness and practicality of the tool with social validity measures. Four raters with experience in praise research completed a survey and coded videos (n = 14) of teachers in authentic situations delivering praise. Overall assessment indicated strong reliability between raters with ICC(2, k) of .80: 95% confidence interval (CI) = [0.77, 0.83], F(269, 19906) = 5.1, p < .001, and mean kappa score of .91. Furthermore, high social validity ratings suggest the BSP-OT is a valuable contribution to the field concerning praise research and teacher development. The process of developing the BSP-OT and study findings are presented, with a discussion of implications and suggestions for future research.","PeriodicalId":47652,"journal":{"name":"Journal of Positive Behavior Interventions","volume":"23 1","pages":"17 - 29"},"PeriodicalIF":1.4000,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/1098300720907988","citationCount":"6","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Positive Behavior Interventions","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1177/1098300720907988","RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"EDUCATION, SPECIAL","Score":null,"Total":0}

引用次数: 6

Abstract

Supported by decades of research on praise and its effect on student behaviors, we developed the Behavior-Specific Praise–Observation Tool (BSP-OT) to measure characteristics of effective praise. We evaluated interrater reliability of the BSP-OT to measure praise specificity, contingency, and variety using intraclass correlation (ICC) and Cohen’s kappa statistics. In addition, we assessed usefulness and practicality of the tool with social validity measures. Four raters with experience in praise research completed a survey and coded videos (n = 14) of teachers in authentic situations delivering praise. Overall assessment indicated strong reliability between raters with ICC(2, k) of .80: 95% confidence interval (CI) = [0.77, 0.83], F(269, 19906) = 5.1, p < .001, and mean kappa score of .91. Furthermore, high social validity ratings suggest the BSP-OT is a valuable contribution to the field concerning praise research and teacher development. The process of developing the BSP-OT and study findings are presented, with a discussion of implications and suggestions for future research.

查看原文本刊更多论文

评价特征观察工具的可靠性评估

在数十年来对表扬及其对学生行为影响的研究的支持下，我们开发了行为特异性表扬-观察工具（BSP-OT）来衡量有效表扬的特征。我们使用组内相关性（ICC）和Cohen’s kappa统计来评估BSP-OT的评估者间可靠性，以衡量表扬的特异性、偶然性和多样性。此外，我们通过社会有效性测量评估了该工具的有用性和实用性。四位有表扬研究经验的评分者完成了一项调查，并对真实情况下的教师进行了视频编码（n=14）。总体评估表明，ICC（2，k）为.80:95%置信区间（CI）=[0.77，0.83]，F（26919906）=5.1，p<.001，平均kappa评分为.91的评分者之间具有很强的可靠性。此外，高社会有效性评级表明，BSP-OT对表扬研究和教师发展领域做出了宝贵贡献。介绍了BSP-OT的开发过程和研究结果，并讨论了对未来研究的启示和建议。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Positive Behavior Interventions Multiple-

CiteScore

4.90

自引率

14.30%

发文量

期刊介绍： ...offers sound, research-based principles of positive behavior support for use in school, home and community settings with people with challenges in behavioral adaptation. Regular features include empirical research; discussion, literature reviews, and conceptual papers; programs, practices, and innovations; forum; and media reviews.