Biases in expert judgements in large-scale S&T Delphi Surveys: How to cope with them?

Abstract

When preparing to a new large-scale science and technology (S&T) Foresight study we decided to analyse available previous Delphi surveys in order to assess the results they achieve. The most important issue for us was how to avoid biases in expert judgements, in particular related to reaching convergence in the second round of the survey compared to the first round. This article presents the results, which turned out to be unexpected for us.

The paper is devoted to analyzing the key characteristics of the Delphi method used in large-scale S&T Foresight studies. It provides insight into the major features of the method, its strong and weak points. A closer analysis is paid to potential constraints of using the Delphi method related to expert judgement biases and approaches to cope with them. There were identified three major groups of typical biases in expert judgements in Delphi studies; for each of them a number of approaches proposed to minimize the biases.

Based on analysis of five large-scale national S&T Delphi surveys (conducted in Japan, Germany, UK, and Russia), it was shown that there is a systemic bias in judgements given by experts in their assessment of importance of topics in the second round compared to the first round. It is also shown that in most cases the variance of expert's judgements in the second round of the survey is higher than in the first round, which means that the second round both gives a systemic bias of judgements and in general does not lead to convergence of expert opinions. It leads to a discussion on the approaches and practical instruments, which can increase the quality of Delphi outputs and make them more useful for policy-making.