A Robustness Test Protocol for Applied QCA: Theory and R Software Application

Abstract

The robustness of qualitative comparative analysis (QCA) results features high on the agenda of methodologists and practitioners. This article aims at advancing this debate on several fronts. First, in line with the extant literature, we take a comprehensive view on robustness arguing that decisions on calibration, consistency, and frequency thresholds should all be tested. Second, we introduce the notion of “sensitivity range” as the range of values for any of these parameters within which the solution formula remains unchanged. Third, we argue that interpreting robustness is more intricate than simply checking if solutions remain unchanged. Beyond sensitivity ranges, researchers should assess robustness by evaluating changes in parameters of fit and the classification of cases as robust, shaky, or possible. Fourth, we enable researchers to perform more than one robustness test at a time by proposing the notions of a “test set”: the overlap between conceptually plausible alternative solutions that can be generated; and of a “robust core”: that part of a QCA solution that withstands the robustness checks. Fifth, we present functionalities implemented in the R package SetMethods that enable researchers to put in practice our proposals. These advancements are integrated into a comprehensive QCA Robustness Test Protocol consisting of three main tests: sensitivity ranges, fit-oriented robustness, and case-oriented robustness. We illustrate the protocol’s implementation with an example on high life expectancy across the globe.