Evaluation of CRM homogeneity in cases of insufficient method repeatability: Comparison of Bayesian analysis with substitutes for ANOVA based estimates

Insufficient method repeatability is a problem characterising the evaluation of certified reference materials (CRMs). In investigating the homogeneity studies of 216 certified parameters from 36 CRMs released by the European Commission’s Joint Research Centre (JRC) over the last four years, it was found that in 1/3 of the cases, the method repeatability (s_r) was too high to calculate the standard deviation between units (s_bb) by classical analysis of variance (ANOVA). It was also found that the application of the repeatability requirement stated in the ISO Guide 35:2017 is not feasible since it would require unrealistically low repeatability standard deviations or an impossibly high number of replicates per unit.

Evaluation of the uncertainty of homogeneity (u_bb) as evaluated by ANOVA using both the maximum of s_bb and 0, the maximum of s_bb and u∗_bb, the uncertainty hidden by method repeatability, the maximum of s_bb and s_bb/√n and Bayesian analysis, using both informative and diffuse priors, as well as the standard deviation of the unit means, were compared using simulated homogeneity studies with repeatabilities of 1–8% and s_bb between 0.2 and 2.8%. It was found that using the maximum of s_bb and s_bb/√n as an estimate of u_bb guards against severe underestimation but usually results in a severe overestimation of the between-unit variation. Using the maximum of (s_bb, 0) shows the least average bias but results in a severe underestimation of u_bb in a high fraction of cases. Using the maximum of (s_bb, u∗_bb) limits, but does not completely eliminate cases of a severe underestimation. Also, it leads to average results biased towards high values. For the range of s_bb and s_r investigated, Bayesian analysis performed worse than max (s_bb, u∗_bb) in limiting severe underestimation of u_bb, but limited the average bias towards high results.

A risk-based approach to cases of insufficient method repeatability is proposed where, after evaluating the other contributions to the uncertainty of certified values, the effect of severe over- and underestimation of u_bb is evaluated, and an appropriate approach is chosen based on this analysis.