Prediction in international relations is hard, sometimes: A commentary on Tetlock et al. (2023)

Abstract

Prediction is hard, especially about the future. But not always. Predicting human behavior at the extremes is fairly easy. Within reason, it's quite straightforward to predict what someone will do tomorrow, at least with respect to their day-to-day routine. It's called a “routine” for a reason. At the other extreme, over eons of human existence, it's quite plausible to predict that the continents will reconnect, dramatically altering the current geographic balance of power. Even further out, although humans could well explore the universe and even establish new homes outside of Earth, we also know, at least according to our current knowledge, that the universe will suffer from heat death.

However, those extremes are not what we care about. The relevant time frame, as acknowledged by the Tetlock et al. piece, is between these extremes, say several years or even a few decades from now. On the one hand, examples of amazingly accurate predictions based on long-term forecasts do seem possible. Perhaps the classic example is John Maynard Keynes' Economic Consequences of the Peace. Noting that the Treaty of Versailles had “nothing to make the defeated Central Empires into good neighbors, nothing to stabilize the new States of Europe, nothing to reclaim Russia,” he predicted, quite ominously and perhaps more accurately than even he realized, that “great privation and great risks to society have become unavoidable” (Keynes, 1919, pp. 226 & 255).

And yet, for each prediction that exhibits such accuracy, there many that are, quite frankly, way off. Consider a data rich enterprise in which accurate forecasts are sought after and valued: population growth. Forecasts of population growth over decades are notoriously difficult despite great effort to make them sound. The uncertainty in such forecasts needs to be explicit, because, as demographer Lee (2011, p. 572) observed, “population projections motivate painful decisions about tax increases, benefit cuts, retirement age, and measures to offset global warming, we need careful measures of their uncertainty”.

Rather than “cherry picking” a particularly good or bad prediction from the past, Tetlock et al. provide systematic assessment of medium-term prediction accuracy. Specifically, they offer an assessment of the Expert Political Judgment project, evaluating the forecasts offered by project participants in 2 years, 1988 and 1997. Moreover, rather than considering a range of topics, the authors reassess the experts’ predictive judgments on two “slower moving” topics: stability versus change in national borders, and nuclear-power status. By the year 2022, 25 years had passed since the later set of forecasts and 34 years had passed since the first set of forecasts. This offers ample time for the predictions offered in those years to pan out. If medium term geopolitical forecasting is in any way possible, it will be found here.

What they find encouraging from the perspective of medium term forcecasting is that, in both issue-area domains, the forecasters performed well. The forecasters had correct classifications of over 90%. But there is a twist. Expert forecasters outperformed nonexpert forecasters in nonproliferation domain (especially with respect to the false-positive rate), but not in the border-change domain. Hence, as the authors remark (on page 16), “Expertise failed to translate into accuracy on over half of the questions: those on border-change/secession.” In other words, while forecasting is good, expertise appears overrated.

Of course, the question is why? Why did expertise seem to offer an advantage in the realm of non-proliferation forecasts (such as whether Iran will acquire the bomb in a given time period), but not border change forecasts (such as whether Ukraine and Russia would go to war over their border)?

One explanation is that nuclear proliferation is a topic sufficiently technical in nature that it is almost guaranteed that a well-informed “civilian”, that is, a nonexpert, will be missing key insights. Someone reading the news and receiving updates on Iran's nuclear program would still likely miss or misinterpret critical aspects regarding the progress of their nuclear program. Consistent with the high false-positive rate of the nonexperts in this issue domain, nonexperts overestimate the ease of developing a nuclear weapon.

This stands in contrast to border disputes. The nature of border clashes and border disputes do not appear to require the same level of technical expertise and, hence, it is (relatively) easier for no experts to conceptualize and understand the prospects of a militarized conflict. Moreover, although neither event—a border change or a country acquiring a nuclear weapons—is common, disputes and conflicts over borders are more common than the development of a nuclear program. Hence, there is less opportunity, compared with nuclear programs, for one to acquire a high false-positive rate.

This assessment of the ability of experts and nonexperts to forecast political events is valuable, but it also points to a more fundamental question: are these even the right type of predictions to be evaluating?

The analysts have a theory in their minds that led them to draw their respective inferences. But a good theory will not specify a simple yes or no outcome. It will be conditional: “yes if this, no, if that.” This is why Friedman and Zeckhauser (2012) argued for intelligence analysts to focus on assessing uncertainty, rather than eliminating uncertainty: the world is too complex and history too contingent for specific events to be fully predictable.

Consider how the field of international relations handled the end of the Cold War. Although there was much lamenting and evaluation over why scholars failed to predict the end, including much opprobrium being directed toward a core theoretical framework of international politics, realism, more important were prognostications for what was to come (Lebow, 1994). The predictions were wide ranging, from claims of history being over to the emergence of a new world order. Most infamously, the eminent realist scholar Mearsheimer (1990) predicted that Europe would head “Back to the Future,” meaning a return to war and violence on a scale not seen since 1945. With the Balkan Wars of the 1990s and the current Russo–Ukraine War, Europe as a whole has been far from peaceful since the end of the Cold War. However, the Western and Central European powers did not return to blows and, in that sense, Europe's future did not look like its past. One could say that Mearsheimer's prediction failed.

However, assessing Mearsheimer's prediction as a failure is to miss a critical point: his prediction was conditional. He specified, in the first footnote of the piece, that his argument presumed NATO dissolving: if NATO dissolves, Europe will return to instability. That did not come to pass. Indeed, the opposite transpired, with NATO expanding its membership starting with the former Warsaw Pact countries of Hungary, the Czech Republic, and Poland in 1999. Given his premise of NATO's demise being necessary for Europe to go “Back to the Future,” it seems that his forecast was actually quite accurate.

All of this simply underscores the difficulty of prediction. Tetlock et al. provide a valuable assessment of whether and how accurate predictions can be made on medium-term events. However, future assessments must do more to account for the contingent and conditional nature of prediction. Analysts and policy makers will gain from not simply knowing if an event could occur, but why it could occur.