There is something that has been coming up in my work a lot which is akin to neglecting to explore beta or base rates. As an example, let’s say I notice that some treatment for a disease has resulted in a big decrease in the occurrence of the disease so I think my treatment is working but I forget to go look at what the rate of decline is the general population. Without that, I can’t know if it is just declining on its own.
Another example would be if, let’s say, I am a house flipper and I notice that from 2019 to 2021 I double my profit so I think I have gotten a lot better at flipping houses. But if I don’t look at the increase in sales prices across the industry, I can’t actually draw that conclusion.
Anyway, I keep seeing this come up over and over again so wanted to start a thread on it. Comment below or bring up any new topics also!
This is precisely the reason we launched our academic project (2014) to publish transparent trend following returns: www.40in20out.com We wanted to measure how different managers utilizing similar-but-different techniques were performing compared to a very simple and easily-understood benchmark methodology. This gives us a ton of insight and manager comparison vs. clean Beta is often overlooked, underutilized or misunderstood as investors talk about generating "Alpha". A great topic to explore.
I see this in sales all the time. We grew 50% so we must be killing it. But the market grew at 80% so in reality we are falling behind. Back in my product management days, I liked to compare myself to the tide. A rising tide lifts all ships, and we wanted to be rising faster than the tide.
Edward Tufte once remarked that the single most important question we can ask when presented with data is "compared to what"? Seriously plagues me the number of interactions - in business and in life - that people act surprised and even mildly offended when I respond to any statement with that question. Out of context data = useless at best...and probably misleading / dangerous.
Another example of how important the base rate is: Some activity increases your chances of early mortality from 0.5% to 1.0%. The media might report this as, "This activity doubles your chance of dying early." And while that is true, you still only have a 1% chance of dying early because of the activity.
> ... but I forget to go look at what the rate of decline is the general population.
This is the hidden side of everything as the Freakonomics guys would say. I call it "causal amnesia".
At my company we attempt to avoid causal amnesia by building an AI corpus that reminds us to consider known beta inputs when formulating any conclusions.
This is a mundane example from my company which we laugh about. I am into Life Insurance Sales and January to March is peak business season in India. Company launch initiative for bottom performing Sales people around Dec and boast improvement in their productivity by March end. They forget that the productivity has increased across the spectrum of DMs and not just the engaged DMs.
This is also related to causal structure, or as the old saying correctly goes: correlation does not equal causation. The famous example has it that ice cream sales increase at the same time as drownings, but the two aren't causally related (unless somone's doing something particularly villanous with the ice cream) - the real reason is the increase in temperature, i.e. summer's here.
As a board member of a large behavioral health provider, I listened to the risk manager talk about the falling rate of suicides within our treatment population. Then, I asked what the rate was in the general population and the expected rate within the treatment population for our industry. Unfortunately, the risk managers did not know or seem interested in learning the comparative rates. When I asked the Chief Medical Officer if he knew those rates, he revealed that our rates were higher in both instances and were approaching expected rates.
I assumed the CMO did not initially correct the risk managers because he would have to answer why our rates were higher than expected and what he was doing about it.
This is a great topic. It’s found everywhere. In finance it’s referred to as risk premia - separating a managers returns based on factors. Kahneman has inside vs outside view - which applies to everything. Mauboussin discusses luck vs skill - which is especially important to business. I feel this research comes secondary to Ego. The self attrubtion bias weighs heavy. No one wants to think their success was from outside forces - or at the very least downplay it. But just being born in America highly influences your success in life. Break that down to state, city, family, genes, school, mentors, and your success can largely be attributed to luck. It’s a hard pill to swallow especially for those who were born with privilege and had personal successes.
It’s a great topic to explore but must be assessed with Ego to have any impact, in my opinion.
Contribution is how we think about it in the impact investing side. We ask how likely would a social outcome happen in the absence of the activities an organisation does. Say there is an intended improvement on teen's mental health given the negative impact of social media, would that still happen regardless of what the NGOs, social enterprises, or government policies do?
If one finds out its contribution is low, that could help with the decision-making process. But reality is calculating the baseline can be easier said than done.
Is base rate neglect a result of availability bias? Would readily available base rate information through some intervention change the tendency or are there other biases at work here too (e.g., confirmation bias)?
Annie, I remember in class you mentioned if someone said if they found 70% of successful founders are difficult to deal with, they also need to check what percentage of unsuccessful founders are difficult to deal with.
Roelof Botha (partner at Sequoia) recently did a podcast with Tim Ferris. Ferris asked him what are the traits of successful founders?
Don Valentine, the founder of Sequoia told Botha, “There’s a two-by-two matrix of people we get to invest in. Exceptional, not exceptional, easy to get along with, not so easy to get along with. Your job today, Roelof, is to figure out in which of those four quadrants we normally make money.”
Botha said he found successful founders are in the quadrant exceptional and not so easy to get along with. Then he went on to explain how historically not so easy to get along founders overcame the odds.
But did Botha also check the traits and percentage of failed founders?
I assume Botha and Sequoia would check these things carefully, but maybe they didn't?
From your experience, do hedge funds and VC's still forget to do this?
People have a tendency to overestimate the impact of their personal efforts in the outcome of things vs. other causal factors like base rates. Not sure what this bias is called?
Maybe a decade ago, my friend bought a new yellow Volkswagon bug. Then I noticed there was so many new yellow Volkwagon bugs in Seattle. There weren't, of course. The only thing that changed was my degree of biased observation.
In the last two years, I've had the (mis)occasion to see family members undergo 3 elective surgeries. Each time, doctors did not/could not offer base rates. They spoke using natural language (rarely, often, etc.). I also found both doctors and friends to offer anecdotes, which tended to in the minds of the patients dwarf base-rate data. There's also some framing effect at play. For example, at our last performance appraisal, some high-achieving employees were told they were given, say, an 8% hike but that the org average was 5%. Here, the reference class was selected narrowly. What about organizations in the same industry/space, for example? There are generally details omitted when it comes to base-rate data that's worth checking.
I was just reading a Freddie Deboer article on how administrators misread gains in education as progress. The absolute level of learning goes up for all kids when gaps between groups still exists or even increase.
Interesting concept that I've long grappled with. We work with Australian schools and have always factored in a 30-40% knockback rate based on price alone (ie, they really want to do the work, but can't afford to just yet). Its an arbitrary percentage based on little more than feel, but we've never questioned it because we've continued to grow and mostly rapidly. I'm wondering how the key players in my team can best be engaged in a conversation that challenges this thinking, not because it's wrong, but because we're more interested in truth than validation.
This is precisely the reason we launched our academic project (2014) to publish transparent trend following returns: www.40in20out.com We wanted to measure how different managers utilizing similar-but-different techniques were performing compared to a very simple and easily-understood benchmark methodology. This gives us a ton of insight and manager comparison vs. clean Beta is often overlooked, underutilized or misunderstood as investors talk about generating "Alpha". A great topic to explore.
I see this in sales all the time. We grew 50% so we must be killing it. But the market grew at 80% so in reality we are falling behind. Back in my product management days, I liked to compare myself to the tide. A rising tide lifts all ships, and we wanted to be rising faster than the tide.
Edward Tufte once remarked that the single most important question we can ask when presented with data is "compared to what"? Seriously plagues me the number of interactions - in business and in life - that people act surprised and even mildly offended when I respond to any statement with that question. Out of context data = useless at best...and probably misleading / dangerous.
Another example of how important the base rate is: Some activity increases your chances of early mortality from 0.5% to 1.0%. The media might report this as, "This activity doubles your chance of dying early." And while that is true, you still only have a 1% chance of dying early because of the activity.
> ... but I forget to go look at what the rate of decline is the general population.
This is the hidden side of everything as the Freakonomics guys would say. I call it "causal amnesia".
At my company we attempt to avoid causal amnesia by building an AI corpus that reminds us to consider known beta inputs when formulating any conclusions.
This is a mundane example from my company which we laugh about. I am into Life Insurance Sales and January to March is peak business season in India. Company launch initiative for bottom performing Sales people around Dec and boast improvement in their productivity by March end. They forget that the productivity has increased across the spectrum of DMs and not just the engaged DMs.
This is also related to causal structure, or as the old saying correctly goes: correlation does not equal causation. The famous example has it that ice cream sales increase at the same time as drownings, but the two aren't causally related (unless somone's doing something particularly villanous with the ice cream) - the real reason is the increase in temperature, i.e. summer's here.
As a board member of a large behavioral health provider, I listened to the risk manager talk about the falling rate of suicides within our treatment population. Then, I asked what the rate was in the general population and the expected rate within the treatment population for our industry. Unfortunately, the risk managers did not know or seem interested in learning the comparative rates. When I asked the Chief Medical Officer if he knew those rates, he revealed that our rates were higher in both instances and were approaching expected rates.
I assumed the CMO did not initially correct the risk managers because he would have to answer why our rates were higher than expected and what he was doing about it.
I always ask, "Compared to what?"
This is a great topic. It’s found everywhere. In finance it’s referred to as risk premia - separating a managers returns based on factors. Kahneman has inside vs outside view - which applies to everything. Mauboussin discusses luck vs skill - which is especially important to business. I feel this research comes secondary to Ego. The self attrubtion bias weighs heavy. No one wants to think their success was from outside forces - or at the very least downplay it. But just being born in America highly influences your success in life. Break that down to state, city, family, genes, school, mentors, and your success can largely be attributed to luck. It’s a hard pill to swallow especially for those who were born with privilege and had personal successes.
It’s a great topic to explore but must be assessed with Ego to have any impact, in my opinion.
Love your work and you've inspired me to document more of my thoughts
I've started a substack on life and guides to small town Japan. Would love for feedback
https://hiddenjapan.substack.com/
Contribution is how we think about it in the impact investing side. We ask how likely would a social outcome happen in the absence of the activities an organisation does. Say there is an intended improvement on teen's mental health given the negative impact of social media, would that still happen regardless of what the NGOs, social enterprises, or government policies do?
If one finds out its contribution is low, that could help with the decision-making process. But reality is calculating the baseline can be easier said than done.
Is base rate neglect a result of availability bias? Would readily available base rate information through some intervention change the tendency or are there other biases at work here too (e.g., confirmation bias)?
Annie, I remember in class you mentioned if someone said if they found 70% of successful founders are difficult to deal with, they also need to check what percentage of unsuccessful founders are difficult to deal with.
Roelof Botha (partner at Sequoia) recently did a podcast with Tim Ferris. Ferris asked him what are the traits of successful founders?
Don Valentine, the founder of Sequoia told Botha, “There’s a two-by-two matrix of people we get to invest in. Exceptional, not exceptional, easy to get along with, not so easy to get along with. Your job today, Roelof, is to figure out in which of those four quadrants we normally make money.”
Botha said he found successful founders are in the quadrant exceptional and not so easy to get along with. Then he went on to explain how historically not so easy to get along founders overcame the odds.
But did Botha also check the traits and percentage of failed founders?
I assume Botha and Sequoia would check these things carefully, but maybe they didn't?
From your experience, do hedge funds and VC's still forget to do this?
https://xkcd.com/552/
Pretty much explains it.
People have a tendency to overestimate the impact of their personal efforts in the outcome of things vs. other causal factors like base rates. Not sure what this bias is called?
Maybe a decade ago, my friend bought a new yellow Volkswagon bug. Then I noticed there was so many new yellow Volkwagon bugs in Seattle. There weren't, of course. The only thing that changed was my degree of biased observation.
In the last two years, I've had the (mis)occasion to see family members undergo 3 elective surgeries. Each time, doctors did not/could not offer base rates. They spoke using natural language (rarely, often, etc.). I also found both doctors and friends to offer anecdotes, which tended to in the minds of the patients dwarf base-rate data. There's also some framing effect at play. For example, at our last performance appraisal, some high-achieving employees were told they were given, say, an 8% hike but that the org average was 5%. Here, the reference class was selected narrowly. What about organizations in the same industry/space, for example? There are generally details omitted when it comes to base-rate data that's worth checking.
Cryptocurrency is the example that springs immediately to mind.
I work in cybersecurity so I see this a lot, but it's not quite the same as base rates are more unknown there vs. other industries.
I was just reading a Freddie Deboer article on how administrators misread gains in education as progress. The absolute level of learning goes up for all kids when gaps between groups still exists or even increase.
Interesting concept that I've long grappled with. We work with Australian schools and have always factored in a 30-40% knockback rate based on price alone (ie, they really want to do the work, but can't afford to just yet). Its an arbitrary percentage based on little more than feel, but we've never questioned it because we've continued to grow and mostly rapidly. I'm wondering how the key players in my team can best be engaged in a conversation that challenges this thinking, not because it's wrong, but because we're more interested in truth than validation.