Consumer-photo quality assessment: Challenges and pitfalls in crowdsourcing

We discuss caveats and challenges in crowdsourcing subjective image quality evaluation studies that use real consumer photos with non-simulated distortions. The subtle nature of consumer image artifacts as well as the nuanced quality differences among photos from various consumer devices, necessitate that crowdsourcing studies be designed with extra caution. In this work, we point out certain caveats to look for in the literature, and we draw attention to and discuss the consequences of various design choices on the subjective responses received. The design choices include: 1) stimulus viewing mode, 2) stimulus habituation, 3) study length, and 4) the location of a number of attention items within the study. We show how these design parameters are key to maximizing response correspondence with lab-based responses, and we emphasize to how this differs from tests that utilize simulated image distortions.