Kendall's W and Inter-rater Reliability

Abstract

In this issue, the article entitled, “Improving Learning of Figurative Concepts in Individuals with Blindness: Adopting Teaching Strategies to Enhance Learning Motivation,” reports the results of a statistical test called Kendall’s W. The full name of the test is Kendall’s Coefficient of Concordance, and it is a statistical test that is designed to measure the level of agreement among raters, or a measure of interrater reliability. It ranges from 0 to 1, with 1 being full agreement among raters and 0 being absolutely no agreement among raters. The easiest way to measure how much agreement exists among raters is by looking at the percent agreement, which also ranges between 0 and 1. A more rigorous and more common statistic is Cohen’s Kappa, which calculates the percentage of items the raters agree on but also accounts for the fact that raters might happen to agree by chance on some items. Cohen’s Kappa (k) also ranges from 0 to 1. As with most statistical tests, Kendall’s W comes along with a p value or level of significance. This means that the W statistic can be evaluated both from the point of view of statistical significance as well as the actual size of the effect. This item is similar to one known as a Pearson correlation coefficient, which is a measure of how closely two variables are related. This statistical test also results in an output measure (r) that ranges from 0 to 1 and a significance level. However, Kendall’s W is not a true correlation coefficient and the raw W measure cannot be viewed in exactly the same way that we view the r connection between two variables. However, there is a linear transformation between the two. The equation is