{"title":"Evaluating interrater agreement with intraclass correlation coefficient in SPICE-based software process assessment","authors":"Hyung-Min Park, Ho-Won Jung","doi":"10.1109/QSIC.2003.1319115","DOIUrl":null,"url":null,"abstract":"As software process assessment (SPA) involves a subjective procedure, its reliability is an important issue. Two types of reliability have intensively been investigated in SPA: internal consistency (internal reliability) and interrater agreement (external reliability). This study investigates interrater agreement. Cohen's Kappa coefficient has been a popular measure for estimating interrater agreement. However, the application of Kappa coefficient in certain situations is incorrect due to the \"Kappa Paradoxes\". To cope with the insufficiency of Kappa coefficient, this study applied the intraclass correlation coefficient (ICC) to estimate interrater agreement. The ICC has not been employed in the SPA context. In addition, we examined the stability of the estimated ICC value by using a bootstrap resampling method. Results show that ICC could be applied where the Kappa coefficient could not be applied, but not all cases.","PeriodicalId":145980,"journal":{"name":"Third International Conference on Quality Software, 2003. Proceedings.","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2003-11-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"11","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Third International Conference on Quality Software, 2003. Proceedings.","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/QSIC.2003.1319115","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 11
Abstract
As software process assessment (SPA) involves a subjective procedure, its reliability is an important issue. Two types of reliability have intensively been investigated in SPA: internal consistency (internal reliability) and interrater agreement (external reliability). This study investigates interrater agreement. Cohen's Kappa coefficient has been a popular measure for estimating interrater agreement. However, the application of Kappa coefficient in certain situations is incorrect due to the "Kappa Paradoxes". To cope with the insufficiency of Kappa coefficient, this study applied the intraclass correlation coefficient (ICC) to estimate interrater agreement. The ICC has not been employed in the SPA context. In addition, we examined the stability of the estimated ICC value by using a bootstrap resampling method. Results show that ICC could be applied where the Kappa coefficient could not be applied, but not all cases.