Evaluating the impact of a digitally implemented subjective standard on professional rugby union player management decision-making

Using a pre-post-test design, this study evaluated the impact of implementing a standard on the reliability of player management decision-making within a professional rugby union environment. Five practitioners from a High-Performance Unit (HPU) rated 22 instances of Global Positioning System (GPS)–based external training load information of 14 players across the 2021–2022 season. This rating was whether a peak/trough/normal exposure in load had occurred. The ratings were repeated at four time points (separated by 2 weeks) before (Pre1, Pre2) and after (Post1, Post2) implementing a consensus statement as a subjective standard (using a dashboard) developed previously within the same environment to identify peaks/troughs in player external training loads. Inter-rater agreement between individuals at each voting round was assessed using Light's Kappa, while pre-post-standard intra-rater agreement was determined from Cohen's Kappa (both with 95% confidence intervals). Changes to dashboard usability from implementing the standard were assessed by administering the System Usability Scale to 11 HPU staff at the four time points. Pre-standard moderate inter-rater agreement (Pre1: 0.53 (0.36–0.69), Pre2: 0.60 (0.42–0.77)) increased to almost perfect agreement (Post1: 0.74 (0.57–0.89), Post2: 0.90 (0.79–1)) post-standard. The intra-rater agreement of 2/5 participants was almost perfect post-standard, while it remained within substantial levels for the others. A linear mixed model ( χ2(3) = 8.85, p = 0.03) illustrated a slight increase in dashboard usability after incorporating the standard (Pre1: 84.09, Pre2: 81.36; Post1: 87.73, Post2: 87.27). Overall, the results highlighted that the subjective standard enhanced reliability of practitioner agreement for the selected decision.