Evaluating color difference measures in images

The most well known and widely used method for comparing two homogeneous color samples is the CIEDE2000 color difference formula because of its strong agreement with human perception. However, the formula is unreliable when applied over images and its spatial extensions have shown little improvement compared with the original formula. Hence, researchers have proposed many methods intending to measure color differences (CDs) in natural scene color images. However, these existing methods have not yet been rigorously compared. Therefore, in this work we review and evaluate CD measures with the purpose of answering the question to what extent do state-of-the-art CD measures agree with human perception of CDs in images? To answer the question, we have reviewed and evaluated eight state-of-the-art CD measures on a public image quality database. We found that the CIEDE2000, its spatial extension and the just noticeable CD measure perform well in computing CDs in images distorted by black level shift and color quantization algorithms (correlation higher than 0.8). However, none of the tested CD measures perform well on identifying CDs for the variety of color related distortions tested in this work, e.g., most of the tested CD measures showed a correlation lower than 0.65.