bout de papier, Vol. 29, No. 4 (2016), pp. 42–43
SUPERFORECASTING: THE ART
AND SCIENCE OF PREDICTION,
2001-2014, by Philip E. Tetlock and Dan
Gardner, Signal/McClelland & Stewart,
2015, 352 pp, $35.00
On August 17, 2007, George Soros hosted twenty of Wall Street’s most senior financiers to lunch at his Long Island estate. Earlier in the month, BNP Paribas had frozen withdrawals from three of its funds and central bankers around the world were becoming increasingly worried about overall economic conditions. The discussion among the financiers centred on whether a recession was imminent. Of the twenty-one attendees, nineteen forecast an economic slowdown and a correction in the market. Only two (one of whom was Soros) forecast the global financial crisis – which, of course, was exactly what happened
This story, related by the former journalist and now Minister of International Trade, Chrystia Freeland, in her book Plutocrats: The Rise of the New Global Super-Rich and the Fall of Everyone Else, underscores the difficulty of accurate prediction. Even the experts are often wrong.
But what about non-experts? Can a group of numerate, informed, and open-minded non-specialists develop insights about future events? It’s a theme that has been gaining traction, reflected in Philip E. Tetlock and Dan Gardner’s Superforecasting: The Art and Science of Prediction (2015). Superforecasting was reviewed favourably in The Economist (“Unclouded vision,” September 26, 2015) and the authors contributed a featured article to that magazine’s The World in 2016 (“Keeping score: What you learn from measuring predictions”). Tetlock is a professor at the Wharton School of Business at the University of Pennsylvania and obtained his undergraduate degree at the University of B.C., where he later taught. Gardner is an Ottawa-based journalist and consultant and was hired earlier this year by the Prime Minister’s Office as a senior advisor.
The authors begin with a trenchant critique of forecasters. Building on ng: The Art and
I avid MacDuff
Tetlock’s decades of NEW TERE TONES BESISEELER research, they conclude that when SUPER predictions are measured against actual FORECASTING outcomes, the average political fore- The Art and Science caster is no more of Prediction PHILIP E TETLOCK accurate than a “dart- DAN GARONER throwing chimpanzee.” Even worse, many forecasters are never held to their inaccurate predictions. For instance, business talk show host Larry Kudlow spent much of the first decade of the 2000s forecasting the “[George W.] Bush boom” shortly before the global economic meltdown of 2008. In another indictment of forecasting practices, the authors show that if forecasters are asked to recall their original prediction about a given event after the event occurs, many will adjust an incorrect prediction to align with how things actually turned out. (In this
The authors begin with a trenchant critique of forecasters. Building on Tetlock’s decades of research, they conclude that when predictions are measured against actual outcomes,
the average political forecaster
is no more accurate than a “dart-throwing chimpanzee.” Even worse, many forecasters are never held to
their inaccurate predictions.
reviewer’s own informal conversations since last federal year’s election, it has been striking how many people have said that they predicted the Liberal victory months before, when that party was in a decidedly third-place position in opinion polls.)
On a global scale, the greatest recent crisis in geopolitical forecasting was the failure to find the expected weapons of mass destruction in Iraq. In the aftermath, the U.S. government’s Intelligence Advanced Research Projects Activity (IARPA) funded a series of forecasting “tournaments”
between 2011 and 2015. Tetlock formed a “Good Judgement Project” team of non-experts, recruited from the general public. His team competed against professional intelligence analysts (who had access to classified information) and two university-based teams. The groups were asked to make a series of predictions on precise, time-bound questions, such as “In the next year, will any country withdraw from the eurozone?” Tetlock’s non-expert forecasters won the tournaments.
What are the stand-out qualities of superforecasters? Drawing on the classic division of “hedgehogs” (who shape their thinking around one or two concepts, such as a core political ideology) and “foxes” (who are broader minded), the latter are better predictors. Notably, superforecasters are ready to make adjustments to their forecasts based on new information and, perhaps most important of all, are committed to a “growth mindset,” by which they seek to learn to become better predictors over time. They are also highly numerate (although they may not have formal training in mathematics or statistics) and very comfortable with probabilistic thinking. Superforecasters are even more accurate when working together in teams and following training. (A sixty-minute tutorial of the “Ten Commandments 9/11 – can affect the course of history and adjust the entire foundation on which specific predictions are based
Given these various considerations, in the final analysis, we need to set practical and realistic expectations of super-forecasting. In particular, both parts of the
“There are no certainties in life –
not even death and taxes if we assign a non-zero probability to the invention of technologies that let us
upload the contents of our brains into a cloud-computing network
and the emergence of a future
society so public-spirited and prosperous that the state can be funded with charitable donations.”
subtitle of the book – “the art and science of prediction” – require equal weight. The measurement and analytical techniques of superforecasting may well be scientific.
But knowing which questions to ask and how to use predictions – the beginning and the end of a decision-making process – is a true art.
For an organization such as Global Affairs Canada, which relies on forecasting both implicitly and explicitly every day, the prospects for using superforecasting are very favourable. The new government’s interest in evidence-based policy fits closely with the book’s emphasis on measurement and accuracy. Superforecasting is not about hackneyed clichés of crystal ›alls or sci-fi but rather the deliberative and structured analysis of geopolitical rends. It encourages thinking about a range if possibilities, allowing an organization to ›repare at an operational level for different sutcomes. Members of PAFSO may wish to ake up the Good Judgement Project’s inviation to join its 2016 Classic Geopolitical hallenge and judge for themselves.
David MacDuff is a Foreign Service Officer urrently based in Ottawa. He served in ingapore from 2010-14.
Originally published in bout de papier, Vol. 29, No. 4 (2016), pp. 42–43. Read the rest of this issue →
