The weather, the stock market, next year’s budget or who will win this weekend’s football game – all have one thing in common: predictions and forecasts are made about each. But these aren’t the only things we make predictions about. Our fixation on forecasting has permeated most areas of our life, and we get irritated when events don’t transpire as we thought they would. So can forecasts be better than they are today?

They can. We’ll soon be making superforcasts that are trimmed and realigned with each new piece of information and then analyzed and improved once the forecasted moment has passed. In this book summary, we’ll explore the complex and intriguing art of making the ultimate forecasts.

In this summary of Superforecasting by Philip E. Tetlock and Dan Gardner, you’ll find out

• why the former CEO of Microsoft predicted about the iPhone's market share;
• how a forecaster predicted Yasser Arafat’s autopsy; and
• why groups of forecasters are more successful than individual forecasters.

Forecasting is something we do all the time, whether we’re mapping our next career move or choosing an investment. Essentially, our forecasts reflect our expectations about what the future holds.

Forecasting is limited, though, since minor events can lead to unforeseen consequences.

We live in a complex world where a single person can instigate huge events. Consider the Arab Spring. It all started when one Tunisian street vendor, Mohamed Bouazizi, set himself on fire after being humiliated by corrupt police officers.

There is a theoretical explanation of why it’s difficult to predict such events. It’s called chaos theory (also known as the butterfly effect), and American meteorologist Edward Lorenz explains it thus: in nonlinear systems like the Earth’s atmosphere, even minute changes can have a considerable impact. If the trajectory of the wind shifts by less than a fraction of a degree, the long-term weather patterns can change drastically. Dramatically put: the flap of a butterfly’s wing in Brazil can cause a tornado in Texas.

But we shouldn’t scrap forecasting altogether just because it has its limitations. Take Edward Lorenz’s field, meteorology. Weather forecasts are relatively reliable when made a few days in advance. Why? Because weather forecasters analyze the accuracy of their forecasts after the fact. By comparing their forecast with the actual weather, they improve their understanding of how the weather works.

The problem is, people in other fields usually do not measure the accuracy of their forecasts!

To improve our forecasting, then, we need to work on accuracy and get serious about comparing what we thought would happen with what actually ends up taking place. And that means getting serious about measuring.

Measuring forecasts might seem like a no-brainer: collect the forecasts, judge their accuracy, do the calculations, et voilà! But it isn’t that easy at all.

To ascertain the accuracy of a forecast, you must first understand the meaning of the original forecast. For example, in April, 2007, media outlets reported that Microsoft’s CEO, Steve Ballmer, had made a prediction: the iPhone would fail to win a significant market share. Considering Apple’s size, Ballmer’s forecast seemed ridiculous, and people literally laughed at him. Others highlighted the fact that Apple controlled 42 percent of the US smartphone market, an undeniably significant number.

But let’s take a look at what he actually said.

He said that, indeed, the iPhone might generate a lot of money; however, it would never gain a significant market share in the global cell-phone market (his prediction: between two and three percent). Rather, the software from his company, Microsoft, would come to dominate. And this prediction was, more or less, correct.

According to Garner IT data, iPhone’s global share of mobile phone sales during the third quarter of 2013 sat at around six percent, which is clearly higher than what Ballmer predicted – but it’s not way off. Meanwhile, Microsoft’s software was being used in the majority of cell phones sold worldwide.

Forecasts should also avoid vague language and use numbers for increased precision.

Vague words, such as “could,” “might” or “likely,” are common in forecasting, but research shows that people attach different meanings to words like these. Forecasters should therefore talk about chance as accurately as possible, by using percentages, for instance.

Consider how American intelligence organizations like the NSA and the CIA claimed that Saddam Hussein was hiding weapons of mass destruction, an allegation that turned out, catastrophically, to be untrue. Had these intelligence agencies calculated with precision and used percentages, the United States might never have invaded Iraq. If they had calculated the chance of Iraq having WMDs at 60 percent, there would still be a 40-percent chance that Saddam had none – a rather shaky justification for going to war.

So how do we avoid horrendous errors like what happened with the WMDs? Clearly, more accuracy is needed in our forecasts. Let’s look at some ways of attaining this.

The best way is to keep score.

To do exactly that, the author’s research team established the government-sponsored Good Judgment Project, which had thousands of volunteers sign up and answer more than one million questions during the course of four years. By using scoring, the team hoped to improve prediction accuracy.

Participants answered such questions as "Will the president of Tunisia flee to a cushy exile in the next month?" or "Will the euro fall below $1.20 in the next twelve months?" Each forecaster then rated the probability of the participant’s forecast, adjusted it accordingly after reading relevant news and, when the forecasted time rolled around, assigned each forecast a Brier score, indicating how accurate the forecast was.

The Brier score, named after Glenn W. Brier, is the most commonly used method of measuring a forecast’s accuracy. The lower the score, the more accurate the forecast: a perfect forecast gets a score of zero. A random guess will generate a 0.5 Brier score, while a forecast that is completely wrong will generate a maximum score of 2.0.

Interpreting the Brier score also depends on the question being asked. You might have a Brier score of 0.2, which seems really good – and yet, your forecast might actually be terrible!

Let’s say we’re predicting weather. If the weather is consistently piping hot with blue skies in Phoenix, Arizona, a forecaster could simply predict hot and sunny weather and get a Brier score of zero, which of course is better than 0.2. But, if you got a 0.2 score for predicting weather in Springfield, Missouri – a place with notoriously changeable weather – you’d be hailed as a world-class meteorologist!

Are all superforecasters gifted thinkers with access to top secret information? Not really. So how do they make such accurate predictions about the future?

A superforecaster tackles a question by breaking down seemingly impossible problems into bite-sized sub-problems. This is known as Fermi-style thinking.

The physicist Enrico Fermi, a central figure in the creation of the atomic bomb, estimated with impressive accuracy things like, for example, the number of piano tuners in Chicago, without a scrap of information at his disposal.

He did this by separating the knowable from the unknowable, which is the first step superforecasters take. For instance, when Yasser Arafat, leader of the Palestine Liberation Organization, died from unknown causes, many speculations surfaced that he had been poisoned. Then in 2012, researchers found inordinately high levels of polonium-210 – a lethal radioactive element – on his belongings. This discovery reinforced the idea that he may well have been poisoned, which led to his body being exhumed and examined in France and Switzerland.

As part of the Good Judgment Project, forecasters were asked whether scientists would find elevated levels of polonium in Yasser Arafat’s body.

Bill Flack, a volunteer forecaster, approached the question Fermi-style and broke down the information.

First, Flack found that Polonium decays rapidly, so if Arafat was poisoned, there was a chance that the polonium wouldn’t be detected in his remains, because he’d died in 2004. Flack researched polonium testing and came to the conclusion that it could be detectable. Later, Flack also took into account that Arafat had Palestinian enemies that could have poisoned him and that the postmortem might be contaminated in order to blame Israel for his death. He forecasted the chance of polonium being found in Arafat’s body at 60 percent.

So, Flack ascertained the basics first, before looking for subsequent assumptions, which is precisely what an effective forecaster does.

As every situation is unique, you should avoid jumping the gun and judging a case too quickly. The best way to approach any question is to take an outside view, which means finding out what the base rate is. But what exactly does that mean?

To offer an example, picture an Italian family who lives in a modest house in the United States. The father works as a bookkeeper and the mother has a part-time position at a daycare center. Their household consists of them, their child and the child’s grandmother.

If you were asked what the chances were of this Italian family owning a pet, you might try to get to the answer by immediately latching on to the details of the family or their living situation. But then, you wouldn’t be a superforecaster! A superforecaster wouldn’t look at details first. Instead, she would begin by researching what percentage, or base rate, of American households own a pet. Within a matter of seconds, thanks to Google, you’d find that figure to be 62 percent. This is your outside view.

Having done that, you can now take the inside view. This will give you details that will adjust the base rate accordingly.

In the example of the Italian family, starting with the outside view gives you an initial estimate: there’s about a 62 percent chance that the family owns a pet. Then, you get more specific and adjust that number. For example, you might check the rate of Italian families living in America who own a pet.

The reasoning behind the outside view stems from a concept called anchoring. An anchor is the initial figure, before adjustments are made. If you instead began with the finer details, your prediction is much likelier to be miles away from any anchor or accurate number.

We’ve seen how superforecasters get the process going, but once you’ve made your initial forecast, you can’t just wait and see if you were right. You’ve got to update and modify your judgment based on any new piece of information.

Remember Bill Flack? After he forecasted a 60-percent chance of polonium being detected in Yasser Arafat’s corpse, he kept an eye on the news and updated his forecast whenever he deemed fit.

Then, long after Flack’s first forecast, the Swiss research team reported that additional testing was required and that they would announce the results later.

As Flack had done a lot of research about polonium, he knew that this postponement meant that the team found polonium and that the further tests were required to confirm its source. So Flack upped his forecast to 65 percent.

As it happened, the Swiss team did find polonium in Arafat’s body and Fleck’s final Brier score came to 0.36. Impressive, considering the difficulty of the question.

You must be wary, though. Although new information can help you, it can also misguide you. For example, US research government agency The Intelligence Advanced Research Projects Activity (IARPA) asked whether there would be less Arctic sea ice on September 15, 2014, than there had been the year before. Superforecaster Doug Lorch concluded that there was a 55-percent chance the answer would be yes.

Two days after his estimate, however, Lorch read a month-old report from the Sea Ice Prediction Network that swayed him enough to increase his prediction to 95 percent, a substantial change based on a single report.

When September 15, 2014, finally rolled around, however, there was more Arctic ice than the year before.

Lorch’s first forecast had this as a 45-percent probability, but, after his adjustment, it fell to a mere five percent.

Skillful updating, therefore, requires teasing out subtle details from extraneous information. Don’t be afraid to change your mind, but think twice about whether new information is useful or not!

Perhaps you’ve heard of the term groupthink. Psychologist Irving Janis, who coined the term, hypothesized that people in small groups build team spirit by unconsciously creating shared illusions that interfere with critical thinking. This interference comes from people shying away from confrontation and simply agreeing with one another.

But not conforming is a source of real value. Independent expression and thought are tremendous assets in any team.

So the research team at The Good Judgment Project decided to see if teamwork would influence accuracy. They did this by creating online platforms where forecasters who were assigned to different groups could communicate with one another.     

At the very beginning, the research team shared insights on group dynamics and cautioned the online groups about groupthink. At the end of the first year, the results were in: on average, those who worked in teams were 23 percent more accurate than individuals. The second year, the research team chose to place superforecasters, as opposed to regular forecasters, in groups and found them to vastly outperform the regular groups.

However, the group dynamics were also impacted. One superforecaster, Elaine Rich, wasn’t pleased with the result. Everyone was overly polite, and there was little critical discussion of alternative viewpoints. In an effort to rectify this, the groups put in extra effort to show that they welcomed constructive criticism.

Another way to increase teamwork effectiveness is through precision questioning, which encourages people to rethink their argument. This is nothing new, of course, as great teachers have practiced precision questioning since the time of Socrates.

Precision questioning means getting to the finer details of an argument, like by asking for the definition of a particular term. Even if opinions are polarized on the matter, this questioning reveals the thinking behind the conclusion, which opens it up to further investigation.

The key message in this book:

Superforecasting isn’t restricted to computers or geniuses. It’s a trainable skill that involves evidence-gathering, score-keeping, keeping yourself updated about new information and patience.

Actionable advice:

Staying updated keeps you a cut above the average forecaster.

Superforecasters update themselves on news relevant to their forecasts far more frequently than normal forecasters. One tip for keeping a trained eye on developments is to create alerts for yourself, by using Google Alerts, for instance. These will alert you to any news that mentions the topic at hand as soon as it becomes available.

Suggested further reading: Forecast by Mark Buchanan

Forecast critiques modern economic theory by revealing its major flaws. Physicist Mark Buchanan takes a close look at basic scientific assumptions behind our economic understanding and, with deft analytical skills, he unveils their inaccuracy. In the second part of the book, Buchanan presents a range of scientific discoveries that he believes might eventually help us to improve modern economics.