The effect of assessor expertise on reliability of analysis of video signs associated with concussion in Australian football

IF 3 2区医学 Q1 SPORT SCIENCES

Journal of science and medicine in sport Pub Date : 2025-01-01 DOI:10.1016/j.jsams.2024.08.200

James R.M. McLaren , Michael Makdissi , Stephen J.C. Hearps , Gavin A. Davis

{"title":"The effect of assessor expertise on reliability of analysis of video signs associated with concussion in Australian football","authors":"James R.M. McLaren , Michael Makdissi , Stephen J.C. Hearps , Gavin A. Davis","doi":"10.1016/j.jsams.2024.08.200","DOIUrl":null,"url":null,"abstract":"<div><h3>Objectives</h3><div>To determine whether spotters with medical training and experience in managing concussion have higher inter-rater reliability and accuracy than non-medical personnel when identifying video signs associated with concussion in Australian football.</div></div><div><h3>Design</h3><div>Retrospective cohort study.</div></div><div><h3>Methods</h3><div>Video clips were collected of all impacts potentially resulting in concussion during 2012 and 2013 Australian Football League (AFL) seasons. Raters were divided into medical doctors and a non-medical group comprising allied health practitioners (physiotherapists) and non-medical/non-allied health personnel (performance analysts). Raters assessed 102 randomly selected videos for signs of concussion. The inter-rater reliability was calculated. Sensitivity, specificity, positive and negative predictive values were calculated by comparing the rater responses to the consensus opinion from two highly experienced clinicians with expertise in concussion.</div></div><div><h3>Results</h3><div>No statistically significant difference in inter-rater reliability was observed between the medical doctors and the non-medical group. Both groups demonstrated good to excellent agreement for <em>slow to get up</em>, <em>clutching at head/face</em> and <em>facial injury</em>. Both groups displayed intra-class coefficient >0.55 for <em>no protective action-floppy</em>, <em>loss of responsiveness</em>, and <em>motor incoordination</em>, and displayed lowest agreement for <em>no protective action-tonic posturing</em>, <em>impact seizure</em> and <em>blank/vacant look</em>. No statistically significant difference was found between the groups for sensitivity, specificity, positive and negative predictive values for correctly classifying video signs compared to the expert consensus opinion.</div></div><div><h3>Conclusions</h3><div>After completing sufficient standardised training and testing, medical and non-medical personnel demonstrate comparable reliability in identifying video signs of concussion in professional Australian football and may be suitable for the role of video spotter.</div></div>","PeriodicalId":16992,"journal":{"name":"Journal of science and medicine in sport","volume":"28 1","pages":"Pages 26-32"},"PeriodicalIF":3.0000,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of science and medicine in sport","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1440244024004857","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"SPORT SCIENCES","Score":null,"Total":0}

引用次数: 0

Abstract

Objectives

To determine whether spotters with medical training and experience in managing concussion have higher inter-rater reliability and accuracy than non-medical personnel when identifying video signs associated with concussion in Australian football.

Design

Retrospective cohort study.

Methods

Video clips were collected of all impacts potentially resulting in concussion during 2012 and 2013 Australian Football League (AFL) seasons. Raters were divided into medical doctors and a non-medical group comprising allied health practitioners (physiotherapists) and non-medical/non-allied health personnel (performance analysts). Raters assessed 102 randomly selected videos for signs of concussion. The inter-rater reliability was calculated. Sensitivity, specificity, positive and negative predictive values were calculated by comparing the rater responses to the consensus opinion from two highly experienced clinicians with expertise in concussion.

Results

No statistically significant difference in inter-rater reliability was observed between the medical doctors and the non-medical group. Both groups demonstrated good to excellent agreement for slow to get up, clutching at head/face and facial injury. Both groups displayed intra-class coefficient > 0.55 for no protective action-floppy, loss of responsiveness, and motor incoordination, and displayed lowest agreement for no protective action-tonic posturing, impact seizure and blank/vacant look. No statistically significant difference was found between the groups for sensitivity, specificity, positive and negative predictive values for correctly classifying video signs compared to the expert consensus opinion.

Conclusions

After completing sufficient standardised training and testing, medical and non-medical personnel demonstrate comparable reliability in identifying video signs of concussion in professional Australian football and may be suitable for the role of video spotter.

查看原文本刊更多论文

评估者的专业知识对分析澳大利亚足球运动中与脑震荡有关的视频迹象的可靠性的影响。

目的确定在识别澳大利亚足球比赛中与脑震荡有关的视频迹象时，受过医疗培训并具有脑震荡处理经验的观测员是否比非医务人员具有更高的互评可靠性和准确性：设计：回顾性队列研究：方法：收集 2012 年和 2013 年澳大利亚足球联赛 (AFL) 赛季中可能导致脑震荡的所有撞击的视频剪辑。评分者分为医生和非医疗组，非医疗组包括专职医疗人员（物理治疗师）和非医疗/非专职医疗人员（表现分析师）。评分员对随机抽取的 102 段视频进行了脑震荡迹象评估。计算了评分者之间的可靠性。将评分者的回答与两名经验丰富的脑震荡专业临床医师的一致意见进行比较，计算出敏感性、特异性、阳性预测值和阴性预测值：结果：医生组和非医生组之间的评分者间可靠性没有明显的统计学差异。两组在起立缓慢、紧抓头部/脸部和面部损伤方面的一致性都很好甚至非常好。两组在无保护动作--软弱无力、反应迟钝和运动不协调方面的级内系数均大于 0.55，而在无保护动作--强直姿势、冲击性抽搐和茫然/空洞眼神方面的一致性最低。与专家共识意见相比，各组在正确分类视频征兆的灵敏度、特异性、阳性预测值和阴性预测值方面均无统计学差异：在完成充分的标准化培训和测试后，医务人员和非医务人员在识别澳大利亚职业足球比赛中脑震荡视频征兆方面表现出相当的可靠性，可能适合担任视频观测员。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of science and medicine in sport 医学-运动科学

CiteScore

7.40

自引率

10.00%

发文量

198

审稿时长

48 days

期刊介绍： The Journal of Science and Medicine in Sport is the official journal of Sports Medicine Australia (SMA) and is an an international refereed research publication covering all aspects of sport science and medicine. The Journal considers for publication Original research and Review papers in the sub-disciplines relating generally to the broad sports medicine and sports science fields: sports medicine, sports injury (including injury epidemiology and injury prevention), physiotherapy, podiatry, physical activity and health, sports science, biomechanics, exercise physiology, motor control and learning, sport and exercise psychology, sports nutrition, public health (as relevant to sport and exercise), and rehabilitation and injury management. Manuscripts with an interdisciplinary perspective with specific applications to sport and exercise and its interaction with health will also be considered.