What Statistics Can and Can’t Tell Us About Ourselves

Harold Eddleston, a seventy-seven-year-old from Greater Manchester, was still reeling from a cancer diagnosis he had been given that week when, on a Saturday morning in February, 1998, he received the worst possible news. . He had to face the future alone: ​​his beloved wife died unexpectedly, due to a heart attack.

Eddleston’s daughter, worried about his health, called their family doctor, a respected local man named Harold Shipman. He went home, sat down with his father, took his hand, and spoke to him gently. Pushed for a prognosis on his departure, Shipman quickly replied, “I’m not going to buy him any Easter eggs.” By Wednesday, Eddleston was dead; He was killed by Dr. Shipman.

Harold Shipman is one of the most prolific serial killers in history. In a twenty-three-year career as a mild-mannered and well-liked family doctor, he injected at least two hundred and fifteen of his patients with fatal doses of opiates. He was finally arrested in September, 1998, six months after Eddleston’s death.

David Spiegelhalter, the author of an important and comprehensive new book, “The Art of Statistics” (Basic), was one of the statisticians tasked with the subsequent public inquiry to establish whether the mortality rate of Shipman’s patients should have aroused suspicion earlier. Then a biostatistician at Cambridge, Spiegelhalter found that Shipman’s excess mortality—the number of his elderly patients who died over the course of his career over the number expected by the average physician—was one hundred and seventy-four. you are a woman and forty. -nine men at the time of his arrest. The total corresponds to the number of victims confirmed in the investigation.

A person’s actions, written only in numbers, tell a deep story. They gestured at the unimaginable grief someone had caused. But at what point does too many deaths become too many deaths? How do you distinguish a suspicious anomaly from a run of bad luck? For that matter, how do we know in advance the number of people we expect to die? Each death is preceded by individual circumstances, private stories, and many reasons; what does it mean to pack all the uncertainty into one number?

In 1825, the French Ministry of Justice ordered the creation of a national collection of crime records. It seems to be the first of its kind anywhere in the world – the statistics of every arrest and conviction in the country, divided by region, gathered and ready for analysis. This is the kind of data set we take for granted today, but at the time it was incredibly novel. This was an early moment of Big Data — the first time that mathematical analysis was applied honestly to the chaotic and unpredictable field of human behavior.

Or maybe not so unpredictable. In the early 18-thirties, a Belgian astronomer and mathematician named Adolphe Quetelet analyzed the numbers and discovered a strange pattern. Crime records are shockingly consistent. Every year, regardless of the actions of the courts and prisons, the number of murders, rapes, and robberies reached almost the same total. There is an “awful precision with which crimes reproduce themselves,” Quetelet said. “We know in advance how many individuals will dirty their hands with the blood of others. How many will be fakes, how many poisons.”

For Quetelet, the evidence suggested that there was something deeper to be discovered. He developed the idea of ​​a “Social Physics,” and began to explore the possibility that human life, like the planets, has an underlying mechanistic trajectory. There is something disconcerting about the idea that, amidst the confusions of choice, chance, and circumstance, mathematics can tell us what it is to be human. Yet Quetelet’s general conclusion still stands: at some level, human life can be measured and predicted. We can now predict, with remarkable accuracy, the number of women in Germany who choose to give birth each year, the number of car accidents in Canada, the number of airplane crashes in the entire Southern Hemisphere, even the number of people who visit an emergency room in New York City on Friday night.

In some ways, this is what you would expect from any large, chaotic system. Consider the predictable and quantifiable way gases behave. It may be impossible to track the movement of each individual gas molecule, but uncertainty and chaos at the molecular level disappear when you look at the bigger picture. Similarly, great regularities emerge from our individual unpredictable lives. It’s like we wake up every morning with the chance, that day, of being a murderer, causing a car accident, deciding to propose to our partner, getting fired from our job. “A hypothesis of ‘chance’ contains all the inevitable unpredictability of the world,” writes Spiegelhalter.

But it’s one thing if your goal is to speak in general terms about who we are, as a collective entity. The trouble comes when you try to go the other way—to learn something about us as individuals from our behavior as a collective. And, of course, those answers are often the ones we want the most.

The dangers of making individual predictions from our collective attitudes are aptly demonstrated in a deal struck by French lawyer André-François Raffray in 1965. He agreed to pay a nine-year-old woman of twenty-five hundred francs per month until his death, where he will take his apartment in Arles.

At the time, the average life expectancy for French women was 74.5 years, and Raffray, who was forty-seven, no doubt thought she had negotiated herself a good contract. Unfortunately for him, as Bill Bryson recounts in his new book, “The Body,” the woman was Jeanne Calment, who became the oldest person on record. He lived for thirty-two years after their deal was signed, outliving Raffray, who died at the age of 77. During that time, he paid more than double the market value for an apartment he could not afford. -an.

Raffray learns the hard way that people are often misrepresented. As the mathematician Ian Stewart points out in “Is Dice Playing God?” (Basic), the average man has one breast and one testicle. In large groups, the natural variability of people cancels out, the random zig is countered by the random zag; but that variability means that we cannot speak with certainty about the individual—a fact that has far-reaching consequences.

Every day, millions of people, including David Spiegelhalter, swallow a small white statin pill to reduce the risk of heart attack and stroke. If you are one of those people, and go on to live a long and happy life without ever suffering a heart attack, you have no way of knowing whether your daily statin is responsible or not. if you don’t have a heart attack in the first place. Of a thousand people who took statins for five years, the drugs helped only eighteen to prevent a major heart attack or stroke. And if you find yourself having a heart attack you won’t know if it was delayed by taking a statin. “All I know,” writes Spiegelhalter, “is that it generally benefits a large group of people like me.”

That’s the rule of thumb in preventative medicine: for most individuals, most drugs don’t work. The fact that they produce a collective good makes them worth taking. But this is a pharmaceutical form of Pascal’s wager: you might as well act as if God is real (and believe that the drugs will work for you), because the consequences otherwise outweigh the inconvenience.

There is so much that, on an individual level, we don’t know: why some people can smoke and avoid lung cancer; why one identical twin stays healthy while the other develops a disease like ALS; why some similar children thrive in school while others flounder. Despite the great promises of Big Data, uncertainty remains so abundant that certain human lives remain infinitely unpredictable. Perhaps the most successful prediction engine in the era of Big Data, at least in financial terms, is Amazon’s recommendation algorithm. It is a huge statistical engine that is worth a lot to the company. Again, this is wrong most of the time. “There was no chance or doubt in the course before my son,” said Mr. Dickens’ Dombey, already envisioning the business career young Paul would enjoy. “His path in life was clear and ready, and marked out before he existed.” Paul, alas, died at the age of six.