Another issue with a lot of these studies is that they have extremely low statistical power. In the breast cancer study example, how many people were even in the study? There are a lot of stat sig calculators for online A/B tests, we can use them for this to get a sense of how many people we need in the study. In this case we have a 0.4% "conversion rate" for getting cancer and we want to detect a 25% change (4 vs 5). We need on the order of 65K people in the study to hit 95% confidence, and that's just to detect that it's not noise. I would guess that they didn't enroll 65K people in this study, and these medical studies have a lot of problems with the data as we know from them mostly not replicating.
Another issue with a lot of these studies is that they have extremely low statistical power. In the breast cancer study example, how many people were even in the study? There are a lot of stat sig calculators for online A/B tests, we can use them for this to get a sense of how many people we need in the study. In this case we have a 0.4% "conversion rate" for getting cancer and we want to detect a 25% change (4 vs 5). We need on the order of 65K people in the study to hit 95% confidence, and that's just to detect that it's not noise. I would guess that they didn't enroll 65K people in this study, and these medical studies have a lot of problems with the data as we know from them mostly not replicating.
"This is a common error we make when we're looking at data" really to say this is a common error when people report data, especially media.
Basic statistics isn't a requirement in most high schools, it should be.