Loss functions for optimizing Kappa as the evaluation measure for classifying diabetic retinopathy and prostate cancer images

Quadratic Weighted Kappa (QWK) is a statistic to measure the agreement between two annotators. QWK has been widely used as the evaluation measure for various medical imaging problems, where, the class labels have a natural ordering, e.g., no Diabetic Retinopathy (DR), mild DR, and severe DR. The easiest way to treat the classification problem with these ordinal labels is to consider the problem as a multiclass classification problem and apply the Cross Entropy (CE) loss. However, when applying CE loss the order of the classes becomes meaningless, i.e., the loss will be same if a healthy image is classified into mild DR or severe DR. At the same time, the QWK score will be severely affected if a healthy image is classified into severe DR than mild DR. The most appropriate way to get a better classification score is to optimize the evaluation measure itself. i.e., directly optimize the QWK statistics. However, this optimization may hinder the learning, and may lead to sub-optimal solutions, and therefore, may give lower performance than expected. On the other hand, Ordinal Regression (OR) based approaches also can be used for such problems. The main focus of this work is to investigate which loss function (CE loss, QWK loss or OR loss) is the most appropriate one to the Convolutional Neural Network-based ordinal classification problems, where, QWK is used as the evaluation measure. Experiments on two public datasets, Diabetic Retinopathy and Prostate Cancer, with two different network architectures suggest that directly optimizing QWK is the better choice when small networks are used. On the other hand, we found that for large networks OR based loss function gives better performance.