Evaluating the Interrater Reliability of the COMFORTneo Scale in Infants: Influence of Gestational Age and Rater Experience.

Background: Pain assessment in preterm infants is challenging because of the absence of an objective reference standard, as self-report is not possible. Behavioral tools, including the COMFORTneo scale, are used in NICUs to assess pain and distress. Although interrater reliability (IRR) is generally adequate, its variation across gestational ages and rater experience remains unclear.

Purpose: To examine the influence of gestational age and rater experience on the interrater reliability (IRR) of the COMFORTneo scale (primary aim) and to evaluate overall IRR (secondary aim).

Methods: We conducted 183 paired assessments of preterm infants by 21 graduated NICU nurses and 12 NICU nursing students, stratified into 5 gestational age groups (24-26, 27-29, 30-32, 33-35, ≥36 weeks). IRR was analyzed using intraclass correlation coefficients (ICC) and generalizability theory. Bootstrapping accounted for unequal distributions.

Results: Overall, IRR was excellent (ICC = 0.85), increasing with gestational age from 0.41 (24-26 weeks) to 0.92 (≥36 weeks). Item-level ICCs ranged from 0.39 (body movement) to 0.92 (respiratory response). Nursing students demonstrated slightly higher IRR (ICC = 0.86) compared with experienced nurses (ICC = 0.74). G-theory indicated that item characteristics explained more variance than rater differences (G = 0.56).

Implications for practice: The COMFORTneo scale is reliable in moderately to late preterm infants but less consistent in extremely preterm infants. Structured training, including refresher sessions for graduated NICU nurses, potentially at intervals shorter than 5 years, is recommended.

Implications for research: Future studies should refine ambiguous items and explore complementary tools for extremely preterm infants.