Evaluating AI-Powered Q&A Systems: A Simple Approach to Determining the Need for Expert Ratings.

This paper introduces a simple approach for assessing whether laypeople or AI-based automations can adequately substitute for expert ratings in the evaluation of AI-powered Q&A systems It employs weighted Cohen's Kappa to assess inter-rater reliability, establishing an expert agreement benchmark and comparing this to individual alternative rater-expert agreements. By visualizing these results in an inter-rater reliability matrix, it is a transparent and structured way to determine the adequacy of non-expert raters. Our findings, based on a real project, suggest that laypeople or AI, in some cases, can match or exceed expert agreement, particularly when risk aversion is a factor. The approach can be adapted to different contexts and rating attributes.