It's Not All Relative

How indexing on relative risk while ignoring absolute risk can generate decision errors

Photo by Bernd 📷 Dittrich on Unsplash

In Peter Attia’s bestseller, Outlive, he tells a story that illustrates a common (and costly) decision error we make in interpreting data about relative risk.

Prior to 2002, hormone replacement therapy (HRT) was a standard treatment for adverse symptoms in menopausal women. In 2002, all of that changed with the publication of results from the Women’s Health Initiative Study (WHI) which reported a 24% increase in the risk of breast cancer among a subset of women taking HRT. As Attia points out, these results generated a massive media response, with a flurry of  “headlines all over the world [that] condemned HRT as a dangerous, cancer-causing therapy.”

The study changed the way that people going through menopause and experiencing adverse symptoms were treated. Given that the decision about whether to take exogenous hormones during menopause comes down to an assessment of the trade-off between the symptoms of menopause HRT was designed to treat and the downside risks of that treatment, the 2002 study seemed to really change that calculus. As Attia put it, HRT became “virtually taboo.” And it’s easy to understand why. Who would want HRT, and what doctor would recommend it to a patient, if it meant such a big increase in the risk of breast cancer?

But this response is an error of interpretation, mainly because it doesn’t take into account the underlying absolute risk of breast cancer.

That 24% increase was absolutely true. You can go fact check it. But that fact-checked statistic doesn’t give you all the information you need to make a good decision about HRT. You need to also know the absolute risk of breast cancer to properly situate that data. As Attia pointed out, “nobody seemed to care that the absolute risk increase of breast cancer for women in the study remained minuscule.” Women in the control group receiving no hormone treatment got breast cancer at a rate of 4-per-1,000. The breast cancer rate of women receiving HRT did, indeed, increase… to 5-per-1,000.

That’s a risk of 0.04% without HRT and 0.05% with. Still incredibly rare.

This is a common error we make when we're looking at data, and it drives errors in the decisions we make. We forget to ask, “Out of how many?” (or “What’s the denominator?”) It feels really scary when you see that 24% increase in the risk of breast cancer (from 4 to 5 women). But asking, “Out of how many?” gets you to the absolute risk, in this case five women get breast cancer out of 1000 women who take HRT instead of four out of 1000 who don’t take it.

The frenzy around the 2002 study sunk the common use of HRT. All because no one seemed to care about the denominator. Meanwhile, hot flashes, night sweats, increased heart rate, and anxiety are experienced by 80% of women going through menopause. And other, more serious adverse effects of menopause that are significantly reduced by HRT include coronary heart disease, osteoporosis, stroke, and dementia. Coronary heat disease, for example, is the leading cause of death of women living in developed nations, and mid-level doses of HRT reduce it by approximately 30%.

This is not to say that every women experiencing menopause should be taking HRT. Each individual must make their own cost-benefit analysis. The issue is that without knowing out of how many, the relative risk statistic is misleading. This is particularly true because the fact checks out.

We are bombarded by data all the time. Data that are true. But without the tools to properly situate the data, we run a high risk of misinterpreting the date which leads to avoidable decision errors.

Of course, the HRT example isn’t an isolated case. The misinferences resulting from indexing on relative vs. absolute risk are common and once you understand the importance of asking, “Out of how many?”, you will begin to see the error everywhere.

Alarming headlines abound that can lead us to jump to conclusions about risk in a way that can drive decision errors if we don’t ask the right questions of the data.

Here are just two examples:

(1) Multiple CT scans in childhood can triple the chance of developing brain cancer and leukemia.

CT SCANS INCREASE RISK OF CANCER

CT SCANS ON CHILDREN ‘COULD TRIPLE BRAIN CANCER RISK’

CT SCANS CAN TRIPLE RISK OF BRAIN CANCER, LEUKEMIA

Yes, it is true that in terms of relative risk, the chances of developing brain cancer or leukemia triple in children who have had CT scans. That sounds really scary and makes for a clickable headline.

But asking, “Out of how many?”, offers a very different view. Query the actual findings about CT scans by reading the source paper, and you’ll find that the absolute risk is quite small. That study estimated that the risk would translate into “one additional case of leukemia and one additional brain tumor per 10,000 head CT scans.”

This doesn’t even include consideration of the massive benefits of CT scans lost if the risk influenced decisions about whether or not to get a CT scan. CT scans are an essential, potentially life-saving tool for diagnosing injuries and illnesses in children (include diagnosing cancer), as well as removing the need for anesthesia and sedation.

(2) One drink a day increases breast cancer by 5%.

EVEN LIGHT DRINKERS AT RISK FOR CANCER

70% OF AMERICANS DON’T REALIZE ALCOHOL IS A MAJOR RISK FACTOR FOR CANCER

The headlines about the link between light drinking and breast cancer followed a 2017 statement by the American Society of Clinical Oncology. The ASCO decided that it needed to take a proactive stance, because “minimizing excessive exposure to alcohol has important implications for cancer prevention.” They warned that, even though the greatest risks were clearly for heavy and moderate drinkers, “some cancer risk persists even at low levels of consumption.”

How much risk? Following the ASCO statement, Aaron Carroll, in his New York Times column, “The Upshot,” answered that question. Light drinking was associated with a 4% increase in risk (compared with not drinking). Sounds like a lot. Until you realize what the absolute risk is. Carroll quantified the absolute risk with the example of a 40-year-old woman. She has a 1.45% of developing breast cancer in the next 10 years. If she is a light drinker, using the data cited in the statement, that risk would become 1.51%, an absolute risk increase of 0.06%. In other words, “if 1,667 40-year-old women became light drinkers, one additional person might develop breast cancer. The other 1,666 would see no difference.”

We all need to become better consumers of information and data. This is particularly true for data that survive a fact check. Data that are true but easily misinterpreted are the most difficult for the information consumer. Something as simple as asking, “Out of how many?” can decrease the kind of misinference that leads to these avoidable decision errors.