Hi Annie! Thanks for doing this. I loved your interview with Peter Attia. I am also a physician and started a health tech company where our product is a smart band for the Apple Watch that allows people to track their personal sun exposure. I have been funding this myself, and feel very strongly about helping people in a preventive manner. 80% of health and beauty changes are attributed to sun over exposure, and I feel like technology can help people in a very significant way. Right now, we’re in a position where I would benefit from some market validation that this product will actually be wanted by consumers to get additional investors. . I have practiced dermatology for 23 years and my patients are very interested, but the investors don’t count that towards market validation, and we are pre-revenue. I know it would be helpful to have this product in the world, but I’m not sure about how to prove it. I would love your thoughts on how to decide whether to go forward with something you know can benefit a lot of people but don’t have the proof investors want. Thank you.
Like most entrepreneurial efforts, the monkey feels like getting people to buy, I would imagine the technical part of creating the watch band doesn't carry a lot of tech risk. Why not charge the people who are "interested" upfront. Friends and family are always interested when they don't have to put money down because they don't want to call your baby ugly. If you offer then 50% off if they buy now, suddenly they might be a lot less interested, and now you have better market signal.
1)What are the best decisions that a person can make?Limiting the number of decisions to five?
2)What are some factors that are common when people make bad decisions or even stupid decisions(Like rushing to make the decision when it it not an emergency)
3)What is the best way a person who barely knows anything about decision making can learn decision making and also get better at decision making in real life?
4)What are the best resources(both free and paid) for helping to make better decisions?
5)How can we get better at making split second decisions/decisions under a short period of time and pressure.
6)What were the best decisions you made in your life(if you feel comfrotable sharing them)
7)Who do you think are some great decision makers ?
8)Who is the best decision maker you know ?
9)Who is the best decision maker in the world (living or dead) ?
A judgment is an input into a decision. You make a judgment about how to model a problem, how to model information, what to believe and what not to believe, what options are under consideration, etc. These judgments are all inputs into a decision = the selection of an option.
e.g. when looking at a menu, you make a set of judgments about what dishes look the most appetizing to you, given your preferences. You might ask the waitstaff for their judgments as well, or the people you are dining with. Based on those judgments, you then select an item to order. Choosing the item to order is the decision.
Thx Annie for a generous platform today... 1 idea; how does one frame acting in our own best interests into a reasonable probability description? Generically , if it possible to state it that way. Would one emphasize the lesser or even negative outcomes ( avoid pain ) or instead focus on the reward aspects. can either of those be put into mathematical terms? Thx !!!
Great question. Anything you do is in service of advancing you toward a goal (or balance of goals). So all things can be framed in a currency (we can call the currency utils) which is how much does it advance me toward my goal or cause me to lose ground? You must consider both the gains and losses to come out with a net expected value of any option you are considering.
Hi Annie. I believe part of your PhD thesis was based on knowledge transfer. I saw this research that outlined the challenges and possible methods. Curious to your views.
“ Robert Haskell’s Transfer of Learning. When Detterman began teaching…
I thought it was important to make things as hard as possible for students so they would discover the principles for themselves. I thought the discovery of principles was a fundamental skill that students needed to learn and transfer to new situations. Now I view education, even graduate education, as the learning of information. I try to make it as easy for students as possible. Where before I was ambiguous about what a good paper was, I now provide examples of the best papers from past classes. Before, I expected students to infer the general conclusion from specific examples. Now I provide the general conclusion and support it with specific examples. In general, I subscribe to the principle that you should teach people exactly what you want them to learn in a situation as close as possible to the one in which the learning will be applied. I don’t count on transfer and I don’t try to promote it except by explicitly pointing out where taught skills may be applied.”
I believe he has thrown the baby out with the bath water. It is true that it is difficult to show transfer across domains (although we did) but transfer within domain is achievable. His inversion stops students from even achieving that with domain. And, imo, teaches them what to think vs how to think. I’d need to know more but from your description sounds like I likely am not a fan.
Hi Annie. Upon some reflection, I think there is a case for both. For example, I am not sure, as an engineering manager if my design engineers were taking concepts from another industry and product and applying them in mine. Electrical engineering in cars or home appliances is different than transmission transformers. However, in e-auctions, would a company benefit from your poker experience, Thinking in Bets? Probably. Some lessons are transferable. Thoughts ?
I love your work, insight and information -- thank you so much for putting this out there so we can all benefit personally and socially. Who is your hero?
Thanks for this Q&A Annie! I've really enjoyed reading the community's questions and your replies.
I'd like to ask about your perspective on the decision-making process as an actor within various roles - professional vs. personal life. As an Employee, Parent, Academic, Activist, or Leader.
How do you work through decisions within the different roles and responsibilities of your life?
The process is the same. What changes is what my values are and what my goals are in each area. Decisions are always goals and values driven. Separately, depending on the milieu, i may be weighting other people’s happiness more or less than my own in the decision that I am making.
Also, decisions usually cross categories. A decision about career is also a decision about parenting for example. So I must interpolate between among various goals and values.
Two models for decisons : 1) Science = facts > calculations >- logic = answer certainty - surety. 2) Rest of the world = interpretation > assessments > judgment = opinion - uncetain - confidence (to varying degrees). So many leap to apply science (mistakenly) to the rest of the world rather than master the process of thinking?
With the prevalence of phrases like: “I hate that, or I hate them, or I hate it” ... is it possible people have completely misinterpreted a term like “critical thinking?” Laughs. And do you think the term confirmation bias is now over-used and/or often misused?
Haha. That’s pretty scary. Anyways. Fun stuff. Thanks for the post.
I think many of the biases are overused because many people don't understand what they are :) I also find that a lot of terms are used to justify dismissing other people while not applying the same critical eye to yourself. :(
Thinking on long time horizons... Will AI eventually turn us into expected value machines? And will there be expected value haves and have nots that are based on limited access to data and technology? I realize the latter is probably already the case (as an example I'm thinking of what the volume of transaction data tells Visa). So I guess the question is, Will it get more extreme?
Like seeing the different ways a chess game can be mapped out, I'd love to see the different possible chains of events that can lead to this undoing. And on AI possibly killing us, I believe it.
If you follow Occam's Razor, it could lead you to thinking about being killed by omission of some consideration by an AI, not necessarily that an evil genius AI decides we're obsolete. They could simply be simpler (dumber) than warranted by the responsibility they are given.
I see a gap in the decision making process that needs some attention. Say for instance that you have a decision to make and you triage it to determine if it is impactful enough to warrant a formal decision science approach. I assume that about 80% will fall into the important but not needing a formal decision. It seems to me that simply going with your gut for the rest of your less consequential choices leaves a lot on the table. For my decisions, I add some well-curated “rules of thumb “ to the decision process to help boost the chances for creating better positions. For example, no decisions when tired, angry, or under undue pressure. Sleep on it. Test the waters before you commit. If most decisions are small, making broad improvements across all of your small choices can only result in better positioning over time.
Absolutely! Not warranting a robust approach does not mean go with your gut. First, the triage of the decision itself is a formal decision science approach. Second, only the least consequential decisions should be fully made with gut. It has to really not matter much in the long run. For the rest, category decisions or personal policies is a great approach. I think this is what you mean by rules of thumb: decide in advance how you will decide about a certain category of decisions. For example, I am vegan. That is a category decision that makes it easy for me when I decide what to eat.
Separately, for many decisions you can still apply a storng structure to the choice but just lower the bar for getting to an option that is "good enough." That is a way to speed up the decision without going with your gut.
As a young adult, I had a policy of never getting involved with anything that involved a getaway car, that kept me out of trouble! Clay Christensen's book "How will you measure your life" has a few, one that I adopted was "no just this once" because he found people would do something "just this once" and it was a very slippery slope to divorce/jail/etc.
In Thinking In Bets, you spend some time talking about self-serving bias. I am familiar with this, and in particular in a business decision context. In my working life, I worked for a global Fortune 50 company, and had responsibility for a global staff. As part of my ongoing education, I took a course on international relations at Thunderbird, which opened my eyes to the existence of self-effacing bias in collectivist cultures. As I think about your book, it brings about this question: Is there some difference in approach for those who may have self-effacing, rather than self-serving bias?
Thinking in terms of gambling, I expect the success of Macau says that self-effacing bias does not make you immune to the mental traps and challenges of gambling. Is there some "style" difference (for lack of a better word) or something else you would say to an audience from a self-effacing culture when thinking about better decisions? Would a mix of people from self-serving and self-effacing cultures have an advantage, as part of a decision pod?
I think a mix of people would be good. Different points of view and clashing biases always helps with accuracy.
The approach to either is going to be the same in terms of decision coaching because the goal is accuracy and callibration. It doesn't matter whether or are overconfident or under, whether you are self serving or self effacing, creating feedback loops that help you callibrate with improve no matter which side your bias is on.
It is also true that what appears to be self effacting bais may be cultural signalling. As you said, Macau is VERY successful!!
So a biotech company has 400 K euros cash left that will run till Feb 2024. The idea was to raise money after the next inflection point ( phase 2a supposed to be finished by now). Unfortunately, things take more time than required, there was delay in recruiting 6 patients out of 12, it means that the Phase 2a clinical trial may not even end by Feb 2024, even if it ends, all take time to get the data etc. However cash will end. what thought process should we use to make decisions - 1. I can't convince investors to raise large amount as we won't have study finished, so we can't claim success and a good inflection point 2. raise small amount of money to move forward till study completes ( may be a convertible) as equity can be expensive as investors know we miscalculated and we dont have good buffer. What would you do to make a decision that could have positive asymmetry
I think the february 2024 deadline adds additional complexity to the decisionraising the question of the "value of time". This question seems similar to someone with terminal cancer who is given 1-? years left to live who loves his job but is eligible to retire. End points matter as does sequence. My (heavily biased) view is that an actuarial approach that considers present value needs to be considered here as well
I have two business model options for a new online subscription product... neither of which has anything except third party research. The data is not great on either option. I've started framing our options differently just to increase our luck surface area... but I still need to pick a route. it sucks
I've been making decisions for a long time but never studied how to do it. I tried poker but gave up. I'm going to try again. My wife and I are taking our boys - 6 and 4 - trick or treating. How much autonomy - how much "helicoptering - how far do we go - how much do we check their candy / takings before letting them - how much do we give them - parenting is causing decision fatigue! Lol!
haha! How much helicoptering or free ranging is a choice about your values and risk attitudes. I can't answer this one. But I relate to the fatigue having four of my own!
Most people hate the idea of uncertainty around unfavorable outcomes. But an opposing force to that is that almost everyone underestimates the tails. So if you don't include the tails in your thinking, you can easily end up too risk-seeking. Not because you are risk averse but because you didn't even have the tails in your consideration set.
It depends on what scary means. People general love the adrenaline rush of a scary ride or a scary movie. But people are risk averse in their decision making around choices of what to invest in or what job to take, etc. They worry about the bad outcomes that can come from a new choice and that makes them not want to make the move. I think that is similar to avoiding the scary. So, I guess it depends on what type of scary :)
I've spent my afternoon reading through Biden's AI Executive Order. It actually looks ok, much better than what I expected with several concrete things baked into it, like compute size above which models and datacenters should be reported to the gov and the idea that models will be considered dangerous even if they are depoyed with technical guardrails which (theoretically) prevent users from utilizing their dangerous capabilities
Yeah I'm following him already. He is pretty one-sided though. I also follow Zvi - https://thezvi.substack.com/ He does weekly AI roundups with everything that happened in the field and is extremely knowledgeable about all the people and tech matters in AI
I agree he is one-sided but I like him because I feel like most of what I read is super bullish on AI. So I find him to be a good balance to the mainly AI optimist point of view. Thank you for the recommendation on Zvi. subscribing now!
Hi Annie! Thanks for doing this. I loved your interview with Peter Attia. I am also a physician and started a health tech company where our product is a smart band for the Apple Watch that allows people to track their personal sun exposure. I have been funding this myself, and feel very strongly about helping people in a preventive manner. 80% of health and beauty changes are attributed to sun over exposure, and I feel like technology can help people in a very significant way. Right now, we’re in a position where I would benefit from some market validation that this product will actually be wanted by consumers to get additional investors. . I have practiced dermatology for 23 years and my patients are very interested, but the investors don’t count that towards market validation, and we are pre-revenue. I know it would be helpful to have this product in the world, but I’m not sure about how to prove it. I would love your thoughts on how to decide whether to go forward with something you know can benefit a lot of people but don’t have the proof investors want. Thank you.
Not to hawk my book, but please read the Monkeys and Pedestals/kill criteria chapter of my book, QUIT. I think you’ll find it very helpful.
Great! I will do it! Thank you.
Like most entrepreneurial efforts, the monkey feels like getting people to buy, I would imagine the technical part of creating the watch band doesn't carry a lot of tech risk. Why not charge the people who are "interested" upfront. Friends and family are always interested when they don't have to put money down because they don't want to call your baby ugly. If you offer then 50% off if they buy now, suddenly they might be a lot less interested, and now you have better market signal.
I like that!
I agree! Thank you for the note and follow up!
1)What are the best decisions that a person can make?Limiting the number of decisions to five?
2)What are some factors that are common when people make bad decisions or even stupid decisions(Like rushing to make the decision when it it not an emergency)
3)What is the best way a person who barely knows anything about decision making can learn decision making and also get better at decision making in real life?
4)What are the best resources(both free and paid) for helping to make better decisions?
5)How can we get better at making split second decisions/decisions under a short period of time and pressure.
6)What were the best decisions you made in your life(if you feel comfrotable sharing them)
7)Who do you think are some great decision makers ?
8)Who is the best decision maker you know ?
9)Who is the best decision maker in the world (living or dead) ?
Would you give us a primer on the difference between a decision and a judgment?
A judgment is an input into a decision. You make a judgment about how to model a problem, how to model information, what to believe and what not to believe, what options are under consideration, etc. These judgments are all inputs into a decision = the selection of an option.
e.g. when looking at a menu, you make a set of judgments about what dishes look the most appetizing to you, given your preferences. You might ask the waitstaff for their judgments as well, or the people you are dining with. Based on those judgments, you then select an item to order. Choosing the item to order is the decision.
Thx Annie for a generous platform today... 1 idea; how does one frame acting in our own best interests into a reasonable probability description? Generically , if it possible to state it that way. Would one emphasize the lesser or even negative outcomes ( avoid pain ) or instead focus on the reward aspects. can either of those be put into mathematical terms? Thx !!!
Great question. Anything you do is in service of advancing you toward a goal (or balance of goals). So all things can be framed in a currency (we can call the currency utils) which is how much does it advance me toward my goal or cause me to lose ground? You must consider both the gains and losses to come out with a net expected value of any option you are considering.
Interesting humans are the ones who collect teachers. The best ones possible. Thx for making yourself available to so many.
What are good resources for teaching decision education to children of various ages in a homeschool setting?
Check The Alliance for Decision Education. Lots of resources there!!
Hi Annie. I believe part of your PhD thesis was based on knowledge transfer. I saw this research that outlined the challenges and possible methods. Curious to your views.
“ Robert Haskell’s Transfer of Learning. When Detterman began teaching…
I thought it was important to make things as hard as possible for students so they would discover the principles for themselves. I thought the discovery of principles was a fundamental skill that students needed to learn and transfer to new situations. Now I view education, even graduate education, as the learning of information. I try to make it as easy for students as possible. Where before I was ambiguous about what a good paper was, I now provide examples of the best papers from past classes. Before, I expected students to infer the general conclusion from specific examples. Now I provide the general conclusion and support it with specific examples. In general, I subscribe to the principle that you should teach people exactly what you want them to learn in a situation as close as possible to the one in which the learning will be applied. I don’t count on transfer and I don’t try to promote it except by explicitly pointing out where taught skills may be applied.”
I believe he has thrown the baby out with the bath water. It is true that it is difficult to show transfer across domains (although we did) but transfer within domain is achievable. His inversion stops students from even achieving that with domain. And, imo, teaches them what to think vs how to think. I’d need to know more but from your description sounds like I likely am not a fan.
Hi Annie. Upon some reflection, I think there is a case for both. For example, I am not sure, as an engineering manager if my design engineers were taking concepts from another industry and product and applying them in mine. Electrical engineering in cars or home appliances is different than transmission transformers. However, in e-auctions, would a company benefit from your poker experience, Thinking in Bets? Probably. Some lessons are transferable. Thoughts ?
I love your work, insight and information -- thank you so much for putting this out there so we can all benefit personally and socially. Who is your hero?
Hmmmm. Good question! It would depend on which part of my life. So I’ll just broadly answer my husband :)
Thanks for this Q&A Annie! I've really enjoyed reading the community's questions and your replies.
I'd like to ask about your perspective on the decision-making process as an actor within various roles - professional vs. personal life. As an Employee, Parent, Academic, Activist, or Leader.
How do you work through decisions within the different roles and responsibilities of your life?
The process is the same. What changes is what my values are and what my goals are in each area. Decisions are always goals and values driven. Separately, depending on the milieu, i may be weighting other people’s happiness more or less than my own in the decision that I am making.
Also, decisions usually cross categories. A decision about career is also a decision about parenting for example. So I must interpolate between among various goals and values.
Two models for decisons : 1) Science = facts > calculations >- logic = answer certainty - surety. 2) Rest of the world = interpretation > assessments > judgment = opinion - uncetain - confidence (to varying degrees). So many leap to apply science (mistakenly) to the rest of the world rather than master the process of thinking?
With the prevalence of phrases like: “I hate that, or I hate them, or I hate it” ... is it possible people have completely misinterpreted a term like “critical thinking?” Laughs. And do you think the term confirmation bias is now over-used and/or often misused?
Haha. That’s pretty scary. Anyways. Fun stuff. Thanks for the post.
I think many of the biases are overused because many people don't understand what they are :) I also find that a lot of terms are used to justify dismissing other people while not applying the same critical eye to yourself. :(
Yeah. The dismissal. That a tell if there ever was one. Thanks again. I really love your books.
Thinking on long time horizons... Will AI eventually turn us into expected value machines? And will there be expected value haves and have nots that are based on limited access to data and technology? I realize the latter is probably already the case (as an example I'm thinking of what the volume of transaction data tells Visa). So I guess the question is, Will it get more extreme?
yes. also, AI may kill us. Not actually kidding.
Death is anything but artificial.
Like seeing the different ways a chess game can be mapped out, I'd love to see the different possible chains of events that can lead to this undoing. And on AI possibly killing us, I believe it.
If you follow Occam's Razor, it could lead you to thinking about being killed by omission of some consideration by an AI, not necessarily that an evil genius AI decides we're obsolete. They could simply be simpler (dumber) than warranted by the responsibility they are given.
I see a gap in the decision making process that needs some attention. Say for instance that you have a decision to make and you triage it to determine if it is impactful enough to warrant a formal decision science approach. I assume that about 80% will fall into the important but not needing a formal decision. It seems to me that simply going with your gut for the rest of your less consequential choices leaves a lot on the table. For my decisions, I add some well-curated “rules of thumb “ to the decision process to help boost the chances for creating better positions. For example, no decisions when tired, angry, or under undue pressure. Sleep on it. Test the waters before you commit. If most decisions are small, making broad improvements across all of your small choices can only result in better positioning over time.
Absolutely! Not warranting a robust approach does not mean go with your gut. First, the triage of the decision itself is a formal decision science approach. Second, only the least consequential decisions should be fully made with gut. It has to really not matter much in the long run. For the rest, category decisions or personal policies is a great approach. I think this is what you mean by rules of thumb: decide in advance how you will decide about a certain category of decisions. For example, I am vegan. That is a category decision that makes it easy for me when I decide what to eat.
Separately, for many decisions you can still apply a storng structure to the choice but just lower the bar for getting to an option that is "good enough." That is a way to speed up the decision without going with your gut.
As a young adult, I had a policy of never getting involved with anything that involved a getaway car, that kept me out of trouble! Clay Christensen's book "How will you measure your life" has a few, one that I adopted was "no just this once" because he found people would do something "just this once" and it was a very slippery slope to divorce/jail/etc.
In Thinking In Bets, you spend some time talking about self-serving bias. I am familiar with this, and in particular in a business decision context. In my working life, I worked for a global Fortune 50 company, and had responsibility for a global staff. As part of my ongoing education, I took a course on international relations at Thunderbird, which opened my eyes to the existence of self-effacing bias in collectivist cultures. As I think about your book, it brings about this question: Is there some difference in approach for those who may have self-effacing, rather than self-serving bias?
Thinking in terms of gambling, I expect the success of Macau says that self-effacing bias does not make you immune to the mental traps and challenges of gambling. Is there some "style" difference (for lack of a better word) or something else you would say to an audience from a self-effacing culture when thinking about better decisions? Would a mix of people from self-serving and self-effacing cultures have an advantage, as part of a decision pod?
I think a mix of people would be good. Different points of view and clashing biases always helps with accuracy.
The approach to either is going to be the same in terms of decision coaching because the goal is accuracy and callibration. It doesn't matter whether or are overconfident or under, whether you are self serving or self effacing, creating feedback loops that help you callibrate with improve no matter which side your bias is on.
It is also true that what appears to be self effacting bais may be cultural signalling. As you said, Macau is VERY successful!!
So a biotech company has 400 K euros cash left that will run till Feb 2024. The idea was to raise money after the next inflection point ( phase 2a supposed to be finished by now). Unfortunately, things take more time than required, there was delay in recruiting 6 patients out of 12, it means that the Phase 2a clinical trial may not even end by Feb 2024, even if it ends, all take time to get the data etc. However cash will end. what thought process should we use to make decisions - 1. I can't convince investors to raise large amount as we won't have study finished, so we can't claim success and a good inflection point 2. raise small amount of money to move forward till study completes ( may be a convertible) as equity can be expensive as investors know we miscalculated and we dont have good buffer. What would you do to make a decision that could have positive asymmetry
Ask these questions:
1) what is the probability phase 2a will be successful?
2) What is the payoff if it is successful? (probability of raising new capital, size of round, increase in size of expected exit, etc.)
3) what is the cost of getting to the end of phase 2a?
With all of that, you should be able to calculate whether the bet is worth it.
Thanks a lot Annie!
I think the february 2024 deadline adds additional complexity to the decisionraising the question of the "value of time". This question seems similar to someone with terminal cancer who is given 1-? years left to live who loves his job but is eligible to retire. End points matter as does sequence. My (heavily biased) view is that an actuarial approach that considers present value needs to be considered here as well
I have two business model options for a new online subscription product... neither of which has anything except third party research. The data is not great on either option. I've started framing our options differently just to increase our luck surface area... but I still need to pick a route. it sucks
I hear you but you are exactly right. In your situation, you must increase the influence of luck.
I've been making decisions for a long time but never studied how to do it. I tried poker but gave up. I'm going to try again. My wife and I are taking our boys - 6 and 4 - trick or treating. How much autonomy - how much "helicoptering - how far do we go - how much do we check their candy / takings before letting them - how much do we give them - parenting is causing decision fatigue! Lol!
haha! How much helicoptering or free ranging is a choice about your values and risk attitudes. I can't answer this one. But I relate to the fatigue having four of my own!
I think of scary as I don't know the odds and think they might be unfavorable or at least have a nasty tail.
Most people hate the idea of uncertainty around unfavorable outcomes. But an opposing force to that is that almost everyone underestimates the tails. So if you don't include the tails in your thinking, you can easily end up too risk-seeking. Not because you are risk averse but because you didn't even have the tails in your consideration set.
are people drawn to the thrill of scary or do they avoid it? any thoughts on how people learn from scary events?
It depends on what scary means. People general love the adrenaline rush of a scary ride or a scary movie. But people are risk averse in their decision making around choices of what to invest in or what job to take, etc. They worry about the bad outcomes that can come from a new choice and that makes them not want to make the move. I think that is similar to avoiding the scary. So, I guess it depends on what type of scary :)
I've spent my afternoon reading through Biden's AI Executive Order. It actually looks ok, much better than what I expected with several concrete things baked into it, like compute size above which models and datacenters should be reported to the gov and the idea that models will be considered dangerous even if they are depoyed with technical guardrails which (theoretically) prevent users from utilizing their dangerous capabilities
I love Gary Marcus' Substack for all things AI. https://garymarcus.substack.com/
I think he is really good on this issue and definitely my go-to.
Yeah I'm following him already. He is pretty one-sided though. I also follow Zvi - https://thezvi.substack.com/ He does weekly AI roundups with everything that happened in the field and is extremely knowledgeable about all the people and tech matters in AI
I agree he is one-sided but I like him because I feel like most of what I read is super bullish on AI. So I find him to be a good balance to the mainly AI optimist point of view. Thank you for the recommendation on Zvi. subscribing now!
Same! I pretty much use Gary’s writings as a grounding against the never ending hype train 😭 Zvi is also very grounded which I love