Dorian Zwanzig, Luca Kreibich, Uta Binder, Ute Dietrich
{"title":"Evaluating AI-Powered Q&A Systems: A Simple Approach to Determining the Need for Expert Ratings.","authors":"Dorian Zwanzig, Luca Kreibich, Uta Binder, Ute Dietrich","doi":"10.3233/SHTI251532","DOIUrl":null,"url":null,"abstract":"<p><p>This paper introduces a simple approach for assessing whether laypeople or AI-based automations can adequately substitute for expert ratings in the evaluation of AI-powered Q&A systems It employs weighted Cohen's Kappa to assess inter-rater reliability, establishing an expert agreement benchmark and comparing this to individual alternative rater-expert agreements. By visualizing these results in an inter-rater reliability matrix, it is a transparent and structured way to determine the adequacy of non-expert raters. Our findings, based on a real project, suggest that laypeople or AI, in some cases, can match or exceed expert agreement, particularly when risk aversion is a factor. The approach can be adapted to different contexts and rating attributes.</p>","PeriodicalId":94357,"journal":{"name":"Studies in health technology and informatics","volume":"332 ","pages":"222-226"},"PeriodicalIF":0.0000,"publicationDate":"2025-10-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Studies in health technology and informatics","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.3233/SHTI251532","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
This paper introduces a simple approach for assessing whether laypeople or AI-based automations can adequately substitute for expert ratings in the evaluation of AI-powered Q&A systems It employs weighted Cohen's Kappa to assess inter-rater reliability, establishing an expert agreement benchmark and comparing this to individual alternative rater-expert agreements. By visualizing these results in an inter-rater reliability matrix, it is a transparent and structured way to determine the adequacy of non-expert raters. Our findings, based on a real project, suggest that laypeople or AI, in some cases, can match or exceed expert agreement, particularly when risk aversion is a factor. The approach can be adapted to different contexts and rating attributes.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
