A critical evaluation of 42, large-scale, science and technology foresight Delphi surveys

Large-scale Delphi surveys on technology foresight started in the 1960s and involve an average of about 2000 participants answering, potentially, up to about 450 items. This contrasts sharply with the participation and content of the more common, smaller-scale Delphi surveys. Previously, Belton et al. developed “six steps” to underpin a well-founded and defensible Delphi process and we apply these steps in a novel evaluation of the quality of 42 large-scale technology foresight surveys. Using a detailed analysis of two exemplar studies and a content analysis of all 42 surveys, we explore whether such surveys differ systematically from “traditional” smaller-scale Delphi surveys and, if so, why this may be and what it may mean for the quality of data produced. We conclude that there are some problematic issues within these surveys—to do with (i) data quality in both the numerical summarizing of participant's between-round feedback and in the reporting of final round numerical responses, (ii) the infrequent elicitation of rationales to justify participants' proffered numerical responses, and, when such rationales are elicited, (iii) the between-round summary and presentation of the rationales. We speculate on the reasons for these design differences in the extant large-scale surveys and conclude that extra-survey political influences, such as differing objectives and the need to demonstrate wide-ranging expert participation, may be the underlying cause. We conclude with considerations and recommendations for the design of future large-scale Delphi surveys to enable the underlying process to become better-founded and more defensible to procedural evaluation.