On the boundaries of experimental research on scenario planning: A commentary on Derbyshire et al. (2022)

The authors of this thoughtfully crafted article argue cogently that increased efforts should be taken to strengthen the role of experiments in building an accumulated body of knowledge of scenario planning. While such efforts can foster the emergence of promising research results, it is pertinent to remain cognizant of the realities which put limits on what experiments can contribute to the advancement such knowledge. Many of these realities ensue from the distinctive characteristics of scenario planning as an intervention (or, using the terminology of experimental design, the “treatment” as the independent variable). Such interventions can be carried out in alternative ways to promote desired outcomes (e.g., inducing changes in the participants' mental models). Apart from the intervention, these outcomes also depend on contextual factors of which some may not be under the experimenter's control. For instance, because scenario planning is typically a group activity, the outcomes depend not only the selected scenario method but also on how well the participants are able to communicate with each other, which in turn depends on their linguistic skills, cognitive abilities, and educational background, including familiarity with futures studies.

For starters, one can posit that the variables in terms of which the interventions, outcomes, and contextual factors are characterized should be similar enough to those encountered in the practice of scenario planning. This would be a prerequisite for interpreting experiments from the viewpoint of practice and for inferring tentative generalizations. Without such a correspondence, there is a potential danger that the experimental research would evolve as a semi-independent activity which—despite fostering the emergence of a continuing stream of empirical experiments as such—would have limited impact in informing the work of practitioners who would continue to rely on their accumulated body of expertise and the insights that they have gained from the many sources of information at their disposal, including anecdotal evidence in reported case studies.

There is an inherent challenge in that if the interventions (e.g., variants of scenario processes), their outcomes (e.g., impacts on mental models), and contextual factors (e.g., participants’ level of trust in each other) are specified with a higher level of granularity, it becomes exceedingly laborious to carry out sufficiently many experimental runs to arrive at validated—perhaps statistically significant—conclusions about the likely outcomes of a given scenario approach in a specific planning context.

To illustrate this point, consider a setting in which there are five participants in each scenario group and four alternative interventions to scenario development based on two variables, (i) the number of scenarios (small vs. large) and (ii) the approach to the characterization of uncertainties (quantitative vs. qualitative). Furthermore, assume that the contextual factors are associated with two variables, (iii) level of expertise (students vs. experienced managers) and (iv) the degree to which the participants know each other before the scenario process (no prior collaboration vs. close colleagues).

In this experimental setup, one would need 5 × 2 × 2 × 2 × 2 = 80 participants to obtain a single observation for each of the 16 possible combinations of these four variables. To arrive at statistically significant results, one would need several observations for each of these combinations. Thus, if the aim is to study many methods and multiple contexts subject to the demands of controlled experiments, this combinatorial growth means that the total number of participants to be recruited would quickly become larger than what can be typically accommodated. This difficulty is compounded by the fact that the participants would not be able to take part in more than a single experimental run (i.e., if the participants have already completed a scenario process using one method, this earlier exposure would affect their behaviour when working with another method).

As a result, if controlled experiments are carried out with the aim of producing statistically significant conclusions, the number of variables and their values in specifying the experimental design (e.g., the 16 combinations above) would have to be limited to very few and the level of granularity in characterizing these variables would have to remain relatively coarse. Due to this coarseness, experimental results may be more adequate for uncovering “general patterns”, formulated as statements about what can usually be expected to occur when scenario planning is carried out in such-and-such a way. In contrast, there would be limited possibilities for making reliable predictions about what outcomes will be attained across the full range of the many ways in which the contextual factors do manifest themselves in practice. It may be particularly challenging to study outcomes caused by interactions between two or more variables. By necessity, all the variables that are involved in such interactions would have to be retained to study the impacts that come about due to the joint occurrence of specific combinations of values for two more variables in the experimental design.

On another note, one needs to bear in mind that statistically significant results are just that–statistical. Their predictive power stems from statistical inferences about what can be expected to happen, on average, when a similar intervention is repeated in an essentially similar context. But unless the results are confirmed with an exceptionally high level of significance (in which case they may be so obvious that no experiments are needed to ascertain them), the results do not dictate what the outcome will be for the next observation that is of acute interest. This may diminish the relevance of statistical significance in providing normative guidance for the design for real-life scenario processes which come in many shapes and sizes. There may also be contextual factors that have not been covered in the experimental literature, making it hard to interpret to what extent the findings of related earlier experiments would hold nevertheless.

Challenges in setting the adequate scope of experiments are likely to surface when seeking to give a precise meaning to what is meant by the “outcomes” of scenario planning as well. Technically, it is not straightforward to measure acclaimed outcomes such as “changing the participants' mental models”. Furthermore, in the larger scheme of things, scenario planning is not an end in itself: rather, it is one of the structured approaches that can contribute to the shaping and implementation of more informed strategies, thereby helping organizations prosper in a world in which resources are in short supply and which is either more or less turbulent (see, e.g., Amer et al., 2013; Bunn & Salo, 1993). Crucially, the contribution that scenario planning can make to support organizational survival depends not only on scenario planning but also on the extent to which organizations are faced with such turbulence or can exert influence on it (cf. Vilkkumaa et al., 2018).

One could even hypothesize that in stable and predictable planning contexts, processes of onerous scenario planning could–despite their measurable impacts on the participants' mental models–lead to excessive administrative overheads. Thus, the relative merits of scenario planning cannot be fully evaluated in isolation of the planning contexts it is enacted. For instance, if the oil crisis of the 1970s had not occurred, the Shell scenarios (Wack, 1985) would probably not have become so celebrated. For comprehensiveness, then, the emerging agenda on experimental research should seek to ascertain in what kinds of planning contexts, differentiated by their degree of turbulence for example, scenario planning may be most effective.

The above points on coarseness have parallels to the selection and characterization of uncertainty factors in scenario development. Often, scenarios are built from a few uncertainty factors (e.g., GDP growth) whose possible realizations (e.g., more than 3%) are described using a few verbal descriptions (or intervals, if numerical measurement scales can be associated with uncertainty factors; see Salo et al., 2022; Tosoni et al., 2019). Because in most scenario processes only rather few scenarios are elaborated, there will be a very large number of possible futures that are not explicitly addressed even within the “closed” boundaries set by the uncertainty factors and their realizations. These boundaries can be expanded through deliberate attempts to accommodate extraordinary phenomena, whereby the scenarios and the processual outcomes of scenario planning may become less predictable and less repeatable. One may therefore hypothesize that the more formal approaches to scenario planning may be more amenable for experimental studies, premised on the assumption that they exhibit more regularities than less structured approaches which rely on intuitionist interactions.

The above remarks notwithstanding, I believe that more experimental research on scenario planning is called for. In view of the large variety of methods of scenario planning and the many kinds of contexts in which they are deployed, it may be hard to arrive at general results that hold conclusively, always and everywhere. Still, even modest experimental findings may be highly useful in advising the shaping of scenario processes and the emergence of better “boilerplate designs” that can be instantiated repeatedly across comparable planning contexts.