I’ve been buried in finishing up my dissertation (which I will be defending on June 15th!!). I have a few posts in the hopper so look for those in the next week. In the meantime, wanted to hear from you on what’s on your mind. I’ll kick it off with the topic of my dissertation: counterfactual forecasting. This is a particular type of forecasting where instead of predicting what might happen in the future, you predict what would have happened in the past if things had been different. Think, “What if I had married the one that got away?” or “What if the US had a more muscular response to the annexation of Crimea? Would they still have invaded Ukraine in 2022?” The work I did for the dissertation (along with Phil Tetlock, Barb Mellers, and others) is on training novices to be better at this type of forecasting.
I love the topic of what-ifs forecasts. Would love your thoughts on it. Or on anything else that’s on your mind really!
This sounds so exciting! One of my unscheduled-future-ideas is to write a "historical" fiction book but with a different timeline. It's not a scholarly idea, but I'd love to learn more about thinking about all the ramifications an unintended side effects should one major historical event have a different outcome.
It is funny that you are writing about this. I just completed an assignment yesterday for a Wharton executive education class I am taking with Dr. Platt on neuroscience about counterfactual forecasting.
Good luck on your defense, Annie! (I’m 61 yo and thinking about pursuing a PhD on my thesis for ethical decision making.) I’ve read Adam Phillips’ book, Missing Out: In praise of the unlived life. It’s a sort of “what would have happened had I done/not done what I did (made/not made that decision)?” kind of book. Have you read it?
In my former professional life, I made the ethically valid decision to say “stop” and my world went tilt as a result. However, no patients were harmed, my coworkers weren’t compelled to violate regulations, and I became a vilified outcast.
Given a second chance, would I have said “stop”? Yes, but I wouldn’t recommend someone else do it. Life presents us numerous dilemmas and the decisions we make define us, our integrity, and the friends we have and no longer have.
Any other good references or words of encouragement? My questioned “behavior” has led me to read many of the popular behavioral economics best sellers. --Gregory
Sounds like the basis for the novel, PostWatch: The Redemption of Christopher Columbus by Orson Scott Card. Basic plot is a researchers who use a technology to perform counterfactual forecasting in the past for interventions from the future!
Judea Pearl puts counterfactual thinking on the third (and highest) rung of his Ladder of Causation. In The Science of Why, he sets these out as
1. Observation, to establish correlation
2. Intervention, to test a causal model by doing something to a key component of the system under observation
3. Counterfactuals, to imagine how the system would produce different outcomes under different interventions that, for practical reasons, cannot be actually performed (the event has already happened, but what if things had been different?) or cannot all be performed at once (should we raise taxes or lower them? all taxes or some? which taxes or how much?)
Love this! yes...this is much of how we learn. We imagine how things might have been different to understand what to do in the future. Thanks for recommending Judea Pearl!
Agile software development approaches usually include holding a Retrospective: after something is delivered, the team talks about what went well and what didn't, what could be improved etc. An Agile talk recently introduced me to the new idea of a Pre-spective: a meeting before the work starts. Imagine the delivery has just happened and absolutely everything had gone wrong with it. Now exactly what went wrong? What could have been done to prevent it or manage the risk? Who will take ownership of each thing identified?
Pre-spective exercises work in part because they overcome some of the challenges of what-if propositions. Once someone observes an outcome, they get anchored to it. Separately, we experience something called creeping determinism: the feeling that what was must have been destined to happen. This means that, having observed an outcome, we will tend to think it had to happen the way it did no matter the counterfactual we consider. Separately, we will tend to conform our view to whatever school of thought we belong to. If we are predisposed to think something would have changed things (e.g. being more hawkish) we will make the world fit that point of view.
Doing the exercise before you observe an outcome helps to overcome some of these issues.
I've little doubt that clicks and social media attention is not what you are seeking with your dissertation, but can I suggest a name change to "sliding doors forecasting" if I'm wrong.
Congratulations and good luck with the defense Annie!
A strategy professor of mine at Wharton once posed a question to our class: How can our organization know if we're at an inflection point? Given that so many forecasts rely on the future resembling the past, getting the answer to this question wrong has profound implications on organizational failure. It can be hard to see in the moment - signals may be very conflicting. (He used the Kodak story as a counterfactual, and unveiled a significant amount of detail around why the case *for* digital cameras was not nearly as clear cut as people often think it was; the exercise was meant to teach us that the signals were conflicting and Kodak management was not sure if it was really at an inflection point)
I've still yet to find a solid methodology for addressing the inflection question (other than capital available to pursue strategic options), but would be curious if you have any wiser thoughts!
I once heard someone make the compelling argument that digital cameras did not kill Kodak -- Facebook did. It wasn't the new technology of taking digital pictures that Kodak couldn't compete with, it was the ability to share photos broadly to your friends and family.
How did you train the subjects to become better at (your words) "counter-factual forecasting"? You'd think that knowledge of the Crimea situation, would assist in making any what-if predictions you might want.
It's a lot like asking "what would've happened had Khrushchev launch the Cuban nukes in Oct 1962?". Easy, we'd all die. Of course if you didn't know that nukes kill people, you wouldn't have come to that conclusion.
In other words, all predictions, contemporary or retroactive, are -- for all practical purposes -- counterfactual. What you don't know, you don't know.
I suppose you might conclude that people with more knowledge are better at making accurate predictions about a certain scenario than those with less knowledge on the certain scenario. But we knew that without the results of these studies.
Great question. We use simulated environments so we can actually know what the truth values are for any counterfactual change. Obviously, moving it into real world environments presents challenges that are unique and that is one direction this research might next go.
Your post reminds me of the Black Mirror movie Bandersnatch. It's about an 80s coder who's adapting a fantasy novel (Bandersnatch) into a video game. It has five possible endings, each based on the choices the viewer made during the interactive movie-watching. While this is new for movies, this isn't new at all for video games. In a way, counterfactual forecasting is at the heart of video-game creation. In your research, did you find any tools that help us do counterfactual forecasting?
Separately, a topic on my mind is how to learn and teach decision-making without bearing the real-life consequences and without waiting for real-life situations to test the decision muscle. That's also partly where my earlier question came from. All the best for the defense, Annie!
I love that show! We are using simulations so I think that fits in with what you are saying...learning decision making without the real life consequences?
I can see the practical value of the counterfactual forecasts for two way door decisions-those where you revisit a decision, perhaps changing your decision. Are there other practical uses I’m missing Annie?
The main practical use to to drive better decisions going forward. As an example, we can consider the question of what would have happened if we had had a more muscular response to the annexation of Ukraine. If we determine this would have deterred a full scare invasion, it might change our policy to similar moves in the future. If we release a product with a particular go to market motion and we imagine different motions, that might inform the way we go to market with new products in the future.
This reminded me of when I first read of the term financial aftcasting which I think was coined by Jim Otar. Uses all of the actual market history to give you all possibilities including luck outcomes as my feeble brain understands it.
Predictive models assume that everyone (all causal factors) will keep behaving the same way in the future (such as player strategy); making correlation patterns constant. If that doesn't happen, wouldn't that mean the "base rate" could drift off far enough to invalidate a model?
Or what if your base rate is derived by a wrong time scale -- like global climate models using historic (or cherry-picked) versus geologic scales?
Should selection of a base rate explicitly identify the causal factors affecting/driving it?
And should you use a base rate if you can't explain all factors contributing to it?
So we cover some of that in our training. Selecting the right reference class is, of course, somewhat of an art. You need to get the time scale right (don't look at rates of lung cancer from the 1960s and use those to predict rates today because so many fewer people smoke. This gets down to understanding the causal drivers of the base rates so you can understand if some change, both retrospective or prospective, will create a dislocation. That is absolutely covered in the training.
That being said, while one of our bots does play a mixed strategy (to insert uncertainty), the bots are non-adaptive. The prediction of the types of adaptations the players might make will add a layer of complexity but one that we want to address in future research. Obviously, that is essentially the poker problem since players are adaptive. What adaptations will they make? How will that change their win rate? How would you adapt back. These are all complex and interesting questions.
We started with non-adaptive bots as our first stab because the problem itself is so hard. Our results give us a map to where to go with the research in the future.
As to you other excellent questions about base rates, when making a decision, you are already making a forecast which means you are already implicitly using base rates. Better to choose a reference class or interpolate across several reference classes, make it explicit, make explicit what you believe the causal drivers are, etc. Use that but let other people poke holes in it. Close the feedback loop once you see how the world unfolds to see how close you were in your forecast. That is the only way to get better. You won't be right but you are already doing these things implicitly. So make the process explicit.
Hi Annie. Can’t wait to read it. The burning question was whether your counterfactual training on core concepts were transferable? Did skills, techniques and lessons in decoding a bot for Game x transfer to decoding a bot for Game y? Any insight you can share before the June 15th unveiling?
Hi Brian! Absolutely! we do show transfer for the forecasting questions that don't involve counterfactuals. That is actually super exciting. effect sizes are similar whether you train in goofspiel or not at all domain specific.
Interesting topic. I recently came across the application of constructor theory in organisational transformation. Rather than trying to identify cause/effect through reducing complex systems to a component level, it seeks to create counterfactual landscape in order to identify what is not possible. The idea is to then identify "constructors" that will produce replicable outcomes (think machines and culture). https://www.constructortheory.org/
Concepts of a multiple universe -- a parallel world or multiple universes -- have been proposed by some physicists in an attempt to make hypothetical sense of Einsteinian and quantum concepts and functions. While for now unknown (and perhaps ultimately unknowable), carrying around in one's head the concept of an Earth 2.0 is useful in trying to understand Earth 1.0. Much of what occurs here on Earth 1 is the result of aleatory events, random sequences that determine an outcome. Because of our implicit hindsight bias, it is all too easy for us to ascribe purposeful cause and resulting effect to specific events turning out one way or another. Imaging what might have happened on Earth 2 provides a useful lens in helping to recognize that things are less deterministic than they seem. It also can act as a filter in our thought process, enabling one to better recognize the larger, structural elements that create the framework in which random events can create such outcomes.
There is a fun show called Counterpart that ran for two seasons starting JK Simmons that explores this. One universe split into two where the two sides begin to diverge because of aleatory events. I bet you would enjoy it. It's interesting to ponder the what-if of access to another version of you whose life trajectory ended up completely different.
I like to apply what if scenarios to stock market movement. It seems a lot of the moviement is counter intutive. I wonder if there is intelligence behind that, or is it just random? Could a gambler be successful as a trader, or should we all just buy an index and move on?
There is also the aspect of different time frames and sectors. Some say it is a random walk, but others like Buffet make billions.
I would lov your perspective on that. Hindsight is so clear, but the future is so cloudy.
Just because the market is a random walk (it is), doesn't mean you can't generate alpha (you can). Similarly, if the market is efficient in the long run, it doesn't mean there aren't irrationalities in the short run that you can exploit. Considering what-if scenarios helps to close feedback loops on why things unfolded as they did that can drive future investment decisions. Of course, this requires you to be good at counterfactual forecasting (part of why we did the work we did!)
Looking back has enormous value for strategic thinkers.
What if's could be potent, if played for example, before a new product launch, they might show a flaw in the roll out plan and one that could be fixed, in effect the team is performing post thinking in real time and one where they can engineer the future. Maybe what if thinking could bring about a new job title for innovative companies, Chief Post Thinking Officer.
Love the new title! Yes...this is similar to a pre-mortem or pre-parade where you imagine different outcomes. That being said, having observed an actual outcome (e.g. Russia did invade Ukraine in spring 2022) makes it much harder to accurately forecast these what-if propositions. If you can get good at it, it is a big competitive advantage.
This sounds like MLB teams saying “the games in September count the most” when all 162 matter equally.
About your topic, is it a human trait to focus on “what ifs” even if it is the best possible outcome? Is this more phycology rather that “mathematics” based approach?
It is a human trait to focus on what-ifs mainly when it is a close call. Or a really undesirable outcome that you wish you could have avoided. But we mainly record them for close calls (I almost got in an accident). We don't like to dig into the best possible outcome because what if we figure out it was due to luck or it wasn't actually the best possible outcome. Of course, there is a lot to learn for digging into those. But we don't because in the short run we can only lose to that.
My son missed being all conference golf team by one stroke. And he’s focused on one putt out of 83 shots. Trying to get him to focus on the totality of the round and want to have a better dad talk...not sure it completely fits....but his tone is ‘what if I would’ve made that putt’....
It is so hard to get over that. One way to help is to ask him how he would feel about the putt if he had made the conference golf team. I imagine it wouldn't have bothered him nearly at all.
This brings up an interesting thing about what-if thinking. We ponder counterfactuals mainly only for close calls. Of course, considering how things might have turned out differently given different conditions in the past is important for any type of consequential result, not just close calls. But close calls are when we naturally think about them and, then, not in a particularly productive way since we are just ruminating on one thing that might have been different.
First of all, break a leg on June 15th and let us know how it goes!
How does your work line up with what Dan Pink reports in The Power Of Regret? It's on my biz reading list but haven't gotten to it yet. I'm guessing that you're taking a more systematic approach. Eager to hear more about your research.
Regret can fuel what-if thinking that can drive us to learn. My work is focused on what the concepts are that we can quickly train novices on that will help them make better forecasts given proposed changed to the past. We use a simulated card game played between to bots, show participants a game the bots played, and then ask them to forecast the outcome given changes to certain turns in the game.
The tl;dr is know what the historical averages are/base rates and start your forecasts there because your default should be regression to the mean. And know the bot strategies (player patterns) because if a proposed what-if diverges enough from the bot's normal pattern, the historical averages won't hold anymore.
It seems really simple but that training has a very large effect on forecast accuracy.
This sounds so exciting! One of my unscheduled-future-ideas is to write a "historical" fiction book but with a different timeline. It's not a scholarly idea, but I'd love to learn more about thinking about all the ramifications an unintended side effects should one major historical event have a different outcome.
Hi Annie,
It is funny that you are writing about this. I just completed an assignment yesterday for a Wharton executive education class I am taking with Dr. Platt on neuroscience about counterfactual forecasting.
Best regards,
Chad
Good luck on your defense, Annie! (I’m 61 yo and thinking about pursuing a PhD on my thesis for ethical decision making.) I’ve read Adam Phillips’ book, Missing Out: In praise of the unlived life. It’s a sort of “what would have happened had I done/not done what I did (made/not made that decision)?” kind of book. Have you read it?
In my former professional life, I made the ethically valid decision to say “stop” and my world went tilt as a result. However, no patients were harmed, my coworkers weren’t compelled to violate regulations, and I became a vilified outcast.
Given a second chance, would I have said “stop”? Yes, but I wouldn’t recommend someone else do it. Life presents us numerous dilemmas and the decisions we make define us, our integrity, and the friends we have and no longer have.
Any other good references or words of encouragement? My questioned “behavior” has led me to read many of the popular behavioral economics best sellers. --Gregory
Sounds like the basis for the novel, PostWatch: The Redemption of Christopher Columbus by Orson Scott Card. Basic plot is a researchers who use a technology to perform counterfactual forecasting in the past for interventions from the future!
Judea Pearl puts counterfactual thinking on the third (and highest) rung of his Ladder of Causation. In The Science of Why, he sets these out as
1. Observation, to establish correlation
2. Intervention, to test a causal model by doing something to a key component of the system under observation
3. Counterfactuals, to imagine how the system would produce different outcomes under different interventions that, for practical reasons, cannot be actually performed (the event has already happened, but what if things had been different?) or cannot all be performed at once (should we raise taxes or lower them? all taxes or some? which taxes or how much?)
Love this! yes...this is much of how we learn. We imagine how things might have been different to understand what to do in the future. Thanks for recommending Judea Pearl!
Agile software development approaches usually include holding a Retrospective: after something is delivered, the team talks about what went well and what didn't, what could be improved etc. An Agile talk recently introduced me to the new idea of a Pre-spective: a meeting before the work starts. Imagine the delivery has just happened and absolutely everything had gone wrong with it. Now exactly what went wrong? What could have been done to prevent it or manage the risk? Who will take ownership of each thing identified?
Pre-spective exercises work in part because they overcome some of the challenges of what-if propositions. Once someone observes an outcome, they get anchored to it. Separately, we experience something called creeping determinism: the feeling that what was must have been destined to happen. This means that, having observed an outcome, we will tend to think it had to happen the way it did no matter the counterfactual we consider. Separately, we will tend to conform our view to whatever school of thought we belong to. If we are predisposed to think something would have changed things (e.g. being more hawkish) we will make the world fit that point of view.
Doing the exercise before you observe an outcome helps to overcome some of these issues.
I've little doubt that clicks and social media attention is not what you are seeking with your dissertation, but can I suggest a name change to "sliding doors forecasting" if I'm wrong.
Love it!
Good luck today
Congratulations and good luck with the defense Annie!
A strategy professor of mine at Wharton once posed a question to our class: How can our organization know if we're at an inflection point? Given that so many forecasts rely on the future resembling the past, getting the answer to this question wrong has profound implications on organizational failure. It can be hard to see in the moment - signals may be very conflicting. (He used the Kodak story as a counterfactual, and unveiled a significant amount of detail around why the case *for* digital cameras was not nearly as clear cut as people often think it was; the exercise was meant to teach us that the signals were conflicting and Kodak management was not sure if it was really at an inflection point)
I've still yet to find a solid methodology for addressing the inflection question (other than capital available to pursue strategic options), but would be curious if you have any wiser thoughts!
I once heard someone make the compelling argument that digital cameras did not kill Kodak -- Facebook did. It wasn't the new technology of taking digital pictures that Kodak couldn't compete with, it was the ability to share photos broadly to your friends and family.
How did you train the subjects to become better at (your words) "counter-factual forecasting"? You'd think that knowledge of the Crimea situation, would assist in making any what-if predictions you might want.
It's a lot like asking "what would've happened had Khrushchev launch the Cuban nukes in Oct 1962?". Easy, we'd all die. Of course if you didn't know that nukes kill people, you wouldn't have come to that conclusion.
In other words, all predictions, contemporary or retroactive, are -- for all practical purposes -- counterfactual. What you don't know, you don't know.
I suppose you might conclude that people with more knowledge are better at making accurate predictions about a certain scenario than those with less knowledge on the certain scenario. But we knew that without the results of these studies.
Great question. We use simulated environments so we can actually know what the truth values are for any counterfactual change. Obviously, moving it into real world environments presents challenges that are unique and that is one direction this research might next go.
Interesting article on how Spotify is using Counterfactual Machine Learning Models: https://www.technologyreview.com/2023/04/04/1070885/complex-math-counterfactuals-spotify-predict-finance-healthcare/
Your post reminds me of the Black Mirror movie Bandersnatch. It's about an 80s coder who's adapting a fantasy novel (Bandersnatch) into a video game. It has five possible endings, each based on the choices the viewer made during the interactive movie-watching. While this is new for movies, this isn't new at all for video games. In a way, counterfactual forecasting is at the heart of video-game creation. In your research, did you find any tools that help us do counterfactual forecasting?
Separately, a topic on my mind is how to learn and teach decision-making without bearing the real-life consequences and without waiting for real-life situations to test the decision muscle. That's also partly where my earlier question came from. All the best for the defense, Annie!
I love that show! We are using simulations so I think that fits in with what you are saying...learning decision making without the real life consequences?
It does.
I can see the practical value of the counterfactual forecasts for two way door decisions-those where you revisit a decision, perhaps changing your decision. Are there other practical uses I’m missing Annie?
The main practical use to to drive better decisions going forward. As an example, we can consider the question of what would have happened if we had had a more muscular response to the annexation of Ukraine. If we determine this would have deterred a full scare invasion, it might change our policy to similar moves in the future. If we release a product with a particular go to market motion and we imagine different motions, that might inform the way we go to market with new products in the future.
Hi Annie,
This reminded me of when I first read of the term financial aftcasting which I think was coined by Jim Otar. Uses all of the actual market history to give you all possibilities including luck outcomes as my feeble brain understands it.
Best wishes on your upcoming dissertation Annie!
I'm going to look into that!
Predictive models assume that everyone (all causal factors) will keep behaving the same way in the future (such as player strategy); making correlation patterns constant. If that doesn't happen, wouldn't that mean the "base rate" could drift off far enough to invalidate a model?
Or what if your base rate is derived by a wrong time scale -- like global climate models using historic (or cherry-picked) versus geologic scales?
Should selection of a base rate explicitly identify the causal factors affecting/driving it?
And should you use a base rate if you can't explain all factors contributing to it?
Good luck on your defense!
So we cover some of that in our training. Selecting the right reference class is, of course, somewhat of an art. You need to get the time scale right (don't look at rates of lung cancer from the 1960s and use those to predict rates today because so many fewer people smoke. This gets down to understanding the causal drivers of the base rates so you can understand if some change, both retrospective or prospective, will create a dislocation. That is absolutely covered in the training.
That being said, while one of our bots does play a mixed strategy (to insert uncertainty), the bots are non-adaptive. The prediction of the types of adaptations the players might make will add a layer of complexity but one that we want to address in future research. Obviously, that is essentially the poker problem since players are adaptive. What adaptations will they make? How will that change their win rate? How would you adapt back. These are all complex and interesting questions.
We started with non-adaptive bots as our first stab because the problem itself is so hard. Our results give us a map to where to go with the research in the future.
As to you other excellent questions about base rates, when making a decision, you are already making a forecast which means you are already implicitly using base rates. Better to choose a reference class or interpolate across several reference classes, make it explicit, make explicit what you believe the causal drivers are, etc. Use that but let other people poke holes in it. Close the feedback loop once you see how the world unfolds to see how close you were in your forecast. That is the only way to get better. You won't be right but you are already doing these things implicitly. So make the process explicit.
Some what-if films...
Groundhog Day
Sliding Doors
Back to the Future
Run Lola Run
It's a Wonderful Life! The Terminator is somewhat in that vein also. :)
Hi Annie. Can’t wait to read it. The burning question was whether your counterfactual training on core concepts were transferable? Did skills, techniques and lessons in decoding a bot for Game x transfer to decoding a bot for Game y? Any insight you can share before the June 15th unveiling?
Hi Brian! Absolutely! we do show transfer for the forecasting questions that don't involve counterfactuals. That is actually super exciting. effect sizes are similar whether you train in goofspiel or not at all domain specific.
Dear Dr. Duke. Can’t wait to read your paper as your methodology and results are profound.
I'll shoot it over to you once I am done!
Interesting topic. I recently came across the application of constructor theory in organisational transformation. Rather than trying to identify cause/effect through reducing complex systems to a component level, it seeks to create counterfactual landscape in order to identify what is not possible. The idea is to then identify "constructors" that will produce replicable outcomes (think machines and culture). https://www.constructortheory.org/
Thank you for sharing! Super interesting approach.
Concepts of a multiple universe -- a parallel world or multiple universes -- have been proposed by some physicists in an attempt to make hypothetical sense of Einsteinian and quantum concepts and functions. While for now unknown (and perhaps ultimately unknowable), carrying around in one's head the concept of an Earth 2.0 is useful in trying to understand Earth 1.0. Much of what occurs here on Earth 1 is the result of aleatory events, random sequences that determine an outcome. Because of our implicit hindsight bias, it is all too easy for us to ascribe purposeful cause and resulting effect to specific events turning out one way or another. Imaging what might have happened on Earth 2 provides a useful lens in helping to recognize that things are less deterministic than they seem. It also can act as a filter in our thought process, enabling one to better recognize the larger, structural elements that create the framework in which random events can create such outcomes.
There is a fun show called Counterpart that ran for two seasons starting JK Simmons that explores this. One universe split into two where the two sides begin to diverge because of aleatory events. I bet you would enjoy it. It's interesting to ponder the what-if of access to another version of you whose life trajectory ended up completely different.
Hello Annie:
I like to apply what if scenarios to stock market movement. It seems a lot of the moviement is counter intutive. I wonder if there is intelligence behind that, or is it just random? Could a gambler be successful as a trader, or should we all just buy an index and move on?
There is also the aspect of different time frames and sectors. Some say it is a random walk, but others like Buffet make billions.
I would lov your perspective on that. Hindsight is so clear, but the future is so cloudy.
Thanks,
Howard
Just because the market is a random walk (it is), doesn't mean you can't generate alpha (you can). Similarly, if the market is efficient in the long run, it doesn't mean there aren't irrationalities in the short run that you can exploit. Considering what-if scenarios helps to close feedback loops on why things unfolded as they did that can drive future investment decisions. Of course, this requires you to be good at counterfactual forecasting (part of why we did the work we did!)
Most interesting!
Looking back has enormous value for strategic thinkers.
What if's could be potent, if played for example, before a new product launch, they might show a flaw in the roll out plan and one that could be fixed, in effect the team is performing post thinking in real time and one where they can engineer the future. Maybe what if thinking could bring about a new job title for innovative companies, Chief Post Thinking Officer.
Love the new title! Yes...this is similar to a pre-mortem or pre-parade where you imagine different outcomes. That being said, having observed an actual outcome (e.g. Russia did invade Ukraine in spring 2022) makes it much harder to accurately forecast these what-if propositions. If you can get good at it, it is a big competitive advantage.
This sounds like MLB teams saying “the games in September count the most” when all 162 matter equally.
About your topic, is it a human trait to focus on “what ifs” even if it is the best possible outcome? Is this more phycology rather that “mathematics” based approach?
It is a human trait to focus on what-ifs mainly when it is a close call. Or a really undesirable outcome that you wish you could have avoided. But we mainly record them for close calls (I almost got in an accident). We don't like to dig into the best possible outcome because what if we figure out it was due to luck or it wasn't actually the best possible outcome. Of course, there is a lot to learn for digging into those. But we don't because in the short run we can only lose to that.
My son missed being all conference golf team by one stroke. And he’s focused on one putt out of 83 shots. Trying to get him to focus on the totality of the round and want to have a better dad talk...not sure it completely fits....but his tone is ‘what if I would’ve made that putt’....
It is so hard to get over that. One way to help is to ask him how he would feel about the putt if he had made the conference golf team. I imagine it wouldn't have bothered him nearly at all.
This brings up an interesting thing about what-if thinking. We ponder counterfactuals mainly only for close calls. Of course, considering how things might have turned out differently given different conditions in the past is important for any type of consequential result, not just close calls. But close calls are when we naturally think about them and, then, not in a particularly productive way since we are just ruminating on one thing that might have been different.
First of all, break a leg on June 15th and let us know how it goes!
How does your work line up with what Dan Pink reports in The Power Of Regret? It's on my biz reading list but haven't gotten to it yet. I'm guessing that you're taking a more systematic approach. Eager to hear more about your research.
Regret can fuel what-if thinking that can drive us to learn. My work is focused on what the concepts are that we can quickly train novices on that will help them make better forecasts given proposed changed to the past. We use a simulated card game played between to bots, show participants a game the bots played, and then ask them to forecast the outcome given changes to certain turns in the game.
The tl;dr is know what the historical averages are/base rates and start your forecasts there because your default should be regression to the mean. And know the bot strategies (player patterns) because if a proposed what-if diverges enough from the bot's normal pattern, the historical averages won't hold anymore.
It seems really simple but that training has a very large effect on forecast accuracy.