Judging reliability at wine and water competitions

Abstract Studies suggest the inter-rater reliability of judges at wine competitions is higher than what would be expected by random chance, but lower than what is observed when experts in other fields make judgments specific to their expertise. To further contextualize the (un-) reliability of wine judging while also extending the study of fine water, we examine the inter-rater reliability of judges at an annual international competition for bottled waters. We find that the inter-rater reliability of water judging is generally better than chance and, at best, about the same as the inter-rater reliability of wine judging at some wine competitions. These results suggest that perceptible differences between fine waters exist but are less pronounced than those between fine wines and, also, that aesthetic standards with respect to fine waters exist but are currently less established than those for fine wines.