In Praise of Pedantic Eclecticism: Pitfalls and Opportunities in the Psychology of Judging

Abstract

This Chapter was written for a collection of essays about the psychology of judging. The Chapter examines some methodological aspects related to experimentation of judging. In particular, it addresses the prevailing concern with external validity. Two central points will be made here. First, it is important to acknowledge that all types of validity are intricately intertwined. Attempting to fix one aspect of validity can be overwhelmed by greater compromises in other aspects. For example, a proposal by Fred Schauer to compare the performance of judges to lay people in complex judicial decision making tasks stands to suffer from low construct validity, thus leading to uninterpretable results. To maximize validity, one needs to be pedantic about the design of the entire study. At the same time, the concern with external validity should not be exaggerated. The discrepancy between the experimental environment and real world settings does not automatically bar all applications of findings from the former to the latter. It does, however, require a cautious application and oftentimes a lot of data. When experimental findings do meet the rigorous demands of external validity and context-attentiveness, they can be applied to real world domains, including specialized ones. This possibility opens up the field of judging to a range of methodological approaches and thus offers the benefit of insights originating from diverse scientific perspectives. To demonstrate the possibilities in this regard, the discussion will center on the application of a particular body of research - coherence based reasoning - to judging.