Our hyper-connected, high-speed digital age generates vast amounts of data that can overwhelm or, if wrangled by skilled number crunchers, transform the way we live and do business. Today, no project is complete without the backing of expertly interpreted large data sets.

In big organizations, medicine and social work – even in small businesses and sports agencies – number crunching is used to discover hidden connections between seemingly unrelated things.

In this book summary, you’ll discover how number crunching is the ultimate skill for facilitating decision making, and you’ll be introduced to its many different uses.

In this summary of Super Crunchers by Ian Ayres,You’ll also learn

• how to to invest in good wines;
• how number crunching can detect auction fraud; and
• how data sets shape policies.

Super crunching might sound like something from a video game, but it refers to the statistical analysis of large data sets – and its applications are nearly endless. Let’s start with some surprising examples.

Super crunching can be used to predict the future quality of vintage wine – indispensable knowledge for wine dealers.

A high-quality red wine is an investment because it gets better as time goes on. This means wine dealers need to be able to predict a wine’s future value.

Orley Ashenfelter, an economist and wine enthusiast, has found a way to accurately predict the quality of Bordeaux wines by analyzing historical relationships between weather and the price of wine.

Ashenfelter has found that each centimeter of winter rain raises the expected price of a wine by 0.00117 dollars. His analysis is so accurate that wine dealers now use it to determine whether or not they should buy a wine, and at what price.

Super crunching can also be applied to baseball. Bill James, a baseball writer, has done for baseball what Ashenfelter did for wine.

James challenged the assumption that baseball experts could assess talent simply by watching the players. He postulated that data-based analysis of players’ previous form in the game would be superior – and he was right!

Traditional talent scouts had little faith in Jeremy Brown, an aspiring player who was overweight, but James’s calculations said otherwise. The Oakland A’s decided to sign Brown based on James’s analysis of his past runs and the decision turned out to be a wise one: Jeremy Brown had his major league debut in September 2006.

Most people feel that statistics are rather dry, but what if statistics could help you find true love or detect fraud? Regression, a powerful super crunching technique, can do both of these things.

Regression is about assessing historical data to determine how different factors influence a certain variable of interest. It worked in the wine and baseball examples. Let’s look in more detail at two further ways it can be applied.

Regression can be used to predict future events, such as compatibility in a potential relationship. The dating site eHarmony uses regression to determine which of its users are likely to form long-lasting partnerships.

For eHarmony, the variable is romantic compatibility, and that variable is influenced by a number of emotional, social and cognitive factors within each person. eHarmony uses regression to produce an equation that determines how these factors fit together most effectively. They assess which combinations of personality traits have fit best together in the past, and use that data to predict compatibility in the future.

This is how eHarmony can produce a very accurate statistical analysis of which of their users you’re likely to find romantically compatible.

Regression can also be used to analyze events or patterns in the past, which is crucial for detecting something like fraud.

In the 1990s, con artists began using a new strategy in auctions for public construction projects in New York. Bidders were supposed to place secret bids for the lowest price they were willing to complete a project for, and then the lowest bid would be revealed at a public bid disclosure before being granted the building contract.

But fraudsters started using bribery to find out what the lowest bid was, then placing their own bid just below it. In the end, the author was able to expose the fraud by using regression on the auction data.

Super crunching isn’t just about analyzing the future and past – it’s also changing the medical and business worlds. Randomized testing, a powerful super crunching technique, is already reshaping medical and business research.

Randomized trials are crucial for determining whether or not a medical treatment is effective. Let’s say you wanted to find out whether chemotherapy is more effective than radiation in treating cancer. You’d probably have to control for a number of factors, like the patient’s diet, age or lifestyle habits. Otherwise those small variables could impact the outcome of your research, like if, for example, more people in the chemotherapy treatment group smoked than their counterparts in the radiation group.

To solve this problem, you’d most likely use a large, randomized study because there, patients are randomly assigned to the two different treatment groups. Regarding the patient’s smoking habits, for instance, you could trust that roughly the same proportion of smokers would show up in both the group treated with chemotherapy and the group treated with radiation, as long as those groups were big enough. So in this case randomization makes the two groups “identical” in terms of smoking, allowing you to more confidently gauge the effect of the treatment. That’s why randomized testing has become such a fundamental tool in medical research.

Businesses can even use randomized testing to determine their marketing strategies.

How do you know if one webpage design attracts more customers than another? The company Offermatica uses randomized testing software to answer that by randomly displaying one design or another when users visit your page.

Offermatica’s software can tell you in real time which page design is getting more click-throughs and which is generating more purchases. That kind of randomization can enhance your site’s performance and help you determine which marketing strategies work best.

Want to influence society? Run a randomized experiment! Randomized testing is changing more than our medicine and businesses – it’s changing our public policy too.

Politicians can use data generated through randomized trials to decide what policies to enact. In fact, policy experiments are already underway in a number of countries.

The Move to Opportunity experiment, which is currently being conducted in five American cities, is one such example. It’s designed to help policymakers determine if poor families benefit from receiving housing vouchers – rental assistance that allows them to relocate to middle class neighborhoods.

Researchers will follow the families for ten years from the point when they started receiving the vouchers, measuring the impact vouchers have on areas of their lives, like employment, education, health and crime.

The results still aren’t in, but the data generated from the experiment will provide policymakers with valuable and concrete information about the benefits of moving a family from one neighborhood to another.

Randomized trials can also be used in global economic development projects. The Poverty Action Lab at MIT, for example, uses randomized tests in developing countries to try to fight poverty.

The lab has run dozens of randomized tests on issues highly important to the developing world, like public health measures and micro-lending. The NGO can use small, randomized pilot studies to see if it’s worthwhile to implement the same strategies on a bigger scale or apply them to an entire country.

Randomized testing is reshaping the way politics operates in the world today.

We’ve seen some examples of how super crunching is already reshaping our world, but what role do humans play in all this? Can mathematical equations really tell us more than traditional experts, with their years of experience and human intuition?

The answer is yes. Super crunching outperforms traditional expertise in almost every case.

A psychologist named Paul Meehl started investigating this as early as 1954, with the publication of his book Clinical Versus Statistical Prediction. Meehl researched around 20 empirical studies, and compared the predictions of “clinical” experts with relatively simple statistical models. He researched a wide range of predictions, including how well schizophrenics would respond to electroshock therapy and how prisoners would benefit from parole.

Meehl found that none of the predictions by human experts were better than their statistical counterparts. He completed a similar analysis before his death in 2003, surveying 136 different man-versus-machine studies. Out of the 136 cases, there were only eight where experts were considerably more accurate than statistical models.

So experts are losing out to super crunching in nearly every field. Why?

Well, super crunching is superior to human analysis because all humans are affected by personal bias. That’s why we pay more attention and give more weight to statistically unusual events. People systematically overestimate the probability of extreme deaths, such as murder, for example, because of the way murder is sensationalized in the media.

Most people would assume that keeping a gun in your house puts your children at risk, but a swimming pool is almost 100 times more likely to kill a child than a gun is.

This doesn’t mean that traditional experts have become irrelevant, however. It just means that their roles have changed.

Lots of experts who base their decisions on personal experience and intuition feel threatened by the rise of super crunching. After all, it seems to undermine their work.

Super crunching actually threatens a whole range of professionals. Loan officers, for example, are completely irrelevant now.

Being a loan officer at a bank used to be a fairly high status position. Loan officers were paid well and had the power to decide who did or didn’t qualify for loans. Today, those decisions are made at central offices and based on statistical algorithms.

Looking a customer in the eye and building a trusting relationship with them aren’t very accurate ways of determining whether or not they’ll pay back a loan. Statistical equations are much better for that. In this case, human input just isn’t necessary anymore.

Human experts aren’t useless, however. Instead of forgoing traditional expertise, we can combine it with super crunching to produce even more accurate predictions and analysis.

In fact, traditional experts make much better decisions when they’re provided with data generated through super crunching. And experts still have to determine which variables should be included in statistical analysis. Super crunching can mathematically test a hypothesis, but a human needs to come up with that hypothesis in the first place.

Several studies suggest that traditional expertise is most effective when combined with statistical algorithms. Essentially, expert opinion should be used to bolster the credibility of statistical analysis, not the other way around.

The key message in this book:

Large data sets can be crunched to create meaningful guidance for real-world decisions. Statistics can predict the future and uncover the past in a way that human judgment, clouded by emotions and personal bias, simply can’t. Super Crunchers are already revolutionizing society, and traditional experts will have to adapt to work with super crunching, not against it. Ultimately, statistics are a powerful tool for understanding and reshaping the world in ways that are better for everyone.

Actionable advice:

Collect and crunch your own data.

More and more businesses are starting to generate their own data by running randomized tests – a critical tool for data-driven decision making. Don’t fall behind! For example, you can determine which of two advertising slogans is more effective at catching your customers’ attention by placing them on your company’s website at random and crunching data about click-throughs or the time visitors spend on your page. That information is invaluable.

Suggested further reading: Carrots and Sticks by Ian Ayres

Carrots and Sticks (2010) is the bible of behavior, incentives and self-control. This book summary explain how you can swap out bad habits with rewards, punishments and formal commitments to yourself. You’ll gain the skills necessary to tackle challenges such as losing weight, quitting smoking and saving for retirement.