I’ve been buried in finishing up my dissertation (which I will be defending on June 15th!!). I have a few posts in the hopper so look for those in the next week. In the meantime, wanted to hear from you on what’s on your mind. I’ll kick it off with the topic of my dissertation: counterfactual forecasting. This is a particular type of forecasting where instead of predicting what might happen in the future, you predict what would have happened in the past if things had been different. Think, “What if I had married the one that got away?” or “What if the US had a more muscular response to the annexation of Crimea? Would they still have invaded Ukraine in 2022?” The work I did for the dissertation (along with Phil Tetlock, Barb Mellers, and others) is on training novices to be better at this type of forecasting.
I love the topic of what-ifs forecasts. Would love your thoughts on it. Or on anything else that’s on your mind really!
This sounds so exciting! One of my unscheduled-future-ideas is to write a "historical" fiction book but with a different timeline. It's not a scholarly idea, but I'd love to learn more about thinking about all the ramifications an unintended side effects should one major historical event have a different outcome.
It is funny that you are writing about this. I just completed an assignment yesterday for a Wharton executive education class I am taking with Dr. Platt on neuroscience about counterfactual forecasting.
Good luck on your defense, Annie! (I’m 61 yo and thinking about pursuing a PhD on my thesis for ethical decision making.) I’ve read Adam Phillips’ book, Missing Out: In praise of the unlived life. It’s a sort of “what would have happened had I done/not done what I did (made/not made that decision)?” kind of book. Have you read it?
In my former professional life, I made the ethically valid decision to say “stop” and my world went tilt as a result. However, no patients were harmed, my coworkers weren’t compelled to violate regulations, and I became a vilified outcast.
Given a second chance, would I have said “stop”? Yes, but I wouldn’t recommend someone else do it. Life presents us numerous dilemmas and the decisions we make define us, our integrity, and the friends we have and no longer have.
Any other good references or words of encouragement? My questioned “behavior” has led me to read many of the popular behavioral economics best sellers. --Gregory
Sounds like the basis for the novel, PostWatch: The Redemption of Christopher Columbus by Orson Scott Card. Basic plot is a researchers who use a technology to perform counterfactual forecasting in the past for interventions from the future!
Judea Pearl puts counterfactual thinking on the third (and highest) rung of his Ladder of Causation. In The Science of Why, he sets these out as
1. Observation, to establish correlation
2. Intervention, to test a causal model by doing something to a key component of the system under observation
3. Counterfactuals, to imagine how the system would produce different outcomes under different interventions that, for practical reasons, cannot be actually performed (the event has already happened, but what if things had been different?) or cannot all be performed at once (should we raise taxes or lower them? all taxes or some? which taxes or how much?)
Agile software development approaches usually include holding a Retrospective: after something is delivered, the team talks about what went well and what didn't, what could be improved etc. An Agile talk recently introduced me to the new idea of a Pre-spective: a meeting before the work starts. Imagine the delivery has just happened and absolutely everything had gone wrong with it. Now exactly what went wrong? What could have been done to prevent it or manage the risk? Who will take ownership of each thing identified?
I've little doubt that clicks and social media attention is not what you are seeking with your dissertation, but can I suggest a name change to "sliding doors forecasting" if I'm wrong.
Congratulations and good luck with the defense Annie!
A strategy professor of mine at Wharton once posed a question to our class: How can our organization know if we're at an inflection point? Given that so many forecasts rely on the future resembling the past, getting the answer to this question wrong has profound implications on organizational failure. It can be hard to see in the moment - signals may be very conflicting. (He used the Kodak story as a counterfactual, and unveiled a significant amount of detail around why the case *for* digital cameras was not nearly as clear cut as people often think it was; the exercise was meant to teach us that the signals were conflicting and Kodak management was not sure if it was really at an inflection point)
I've still yet to find a solid methodology for addressing the inflection question (other than capital available to pursue strategic options), but would be curious if you have any wiser thoughts!
How did you train the subjects to become better at (your words) "counter-factual forecasting"? You'd think that knowledge of the Crimea situation, would assist in making any what-if predictions you might want.
It's a lot like asking "what would've happened had Khrushchev launch the Cuban nukes in Oct 1962?". Easy, we'd all die. Of course if you didn't know that nukes kill people, you wouldn't have come to that conclusion.
In other words, all predictions, contemporary or retroactive, are -- for all practical purposes -- counterfactual. What you don't know, you don't know.
I suppose you might conclude that people with more knowledge are better at making accurate predictions about a certain scenario than those with less knowledge on the certain scenario. But we knew that without the results of these studies.
Your post reminds me of the Black Mirror movie Bandersnatch. It's about an 80s coder who's adapting a fantasy novel (Bandersnatch) into a video game. It has five possible endings, each based on the choices the viewer made during the interactive movie-watching. While this is new for movies, this isn't new at all for video games. In a way, counterfactual forecasting is at the heart of video-game creation. In your research, did you find any tools that help us do counterfactual forecasting?
Separately, a topic on my mind is how to learn and teach decision-making without bearing the real-life consequences and without waiting for real-life situations to test the decision muscle. That's also partly where my earlier question came from. All the best for the defense, Annie!
I can see the practical value of the counterfactual forecasts for two way door decisions-those where you revisit a decision, perhaps changing your decision. Are there other practical uses I’m missing Annie?
This reminded me of when I first read of the term financial aftcasting which I think was coined by Jim Otar. Uses all of the actual market history to give you all possibilities including luck outcomes as my feeble brain understands it.
Predictive models assume that everyone (all causal factors) will keep behaving the same way in the future (such as player strategy); making correlation patterns constant. If that doesn't happen, wouldn't that mean the "base rate" could drift off far enough to invalidate a model?
Or what if your base rate is derived by a wrong time scale -- like global climate models using historic (or cherry-picked) versus geologic scales?
Should selection of a base rate explicitly identify the causal factors affecting/driving it?
And should you use a base rate if you can't explain all factors contributing to it?
Hi Annie. Can’t wait to read it. The burning question was whether your counterfactual training on core concepts were transferable? Did skills, techniques and lessons in decoding a bot for Game x transfer to decoding a bot for Game y? Any insight you can share before the June 15th unveiling?
Interesting topic. I recently came across the application of constructor theory in organisational transformation. Rather than trying to identify cause/effect through reducing complex systems to a component level, it seeks to create counterfactual landscape in order to identify what is not possible. The idea is to then identify "constructors" that will produce replicable outcomes (think machines and culture). https://www.constructortheory.org/
Concepts of a multiple universe -- a parallel world or multiple universes -- have been proposed by some physicists in an attempt to make hypothetical sense of Einsteinian and quantum concepts and functions. While for now unknown (and perhaps ultimately unknowable), carrying around in one's head the concept of an Earth 2.0 is useful in trying to understand Earth 1.0. Much of what occurs here on Earth 1 is the result of aleatory events, random sequences that determine an outcome. Because of our implicit hindsight bias, it is all too easy for us to ascribe purposeful cause and resulting effect to specific events turning out one way or another. Imaging what might have happened on Earth 2 provides a useful lens in helping to recognize that things are less deterministic than they seem. It also can act as a filter in our thought process, enabling one to better recognize the larger, structural elements that create the framework in which random events can create such outcomes.
I like to apply what if scenarios to stock market movement. It seems a lot of the moviement is counter intutive. I wonder if there is intelligence behind that, or is it just random? Could a gambler be successful as a trader, or should we all just buy an index and move on?
There is also the aspect of different time frames and sectors. Some say it is a random walk, but others like Buffet make billions.
I would lov your perspective on that. Hindsight is so clear, but the future is so cloudy.
Looking back has enormous value for strategic thinkers.
What if's could be potent, if played for example, before a new product launch, they might show a flaw in the roll out plan and one that could be fixed, in effect the team is performing post thinking in real time and one where they can engineer the future. Maybe what if thinking could bring about a new job title for innovative companies, Chief Post Thinking Officer.
This sounds like MLB teams saying “the games in September count the most” when all 162 matter equally.
About your topic, is it a human trait to focus on “what ifs” even if it is the best possible outcome? Is this more phycology rather that “mathematics” based approach?
My son missed being all conference golf team by one stroke. And he’s focused on one putt out of 83 shots. Trying to get him to focus on the totality of the round and want to have a better dad talk...not sure it completely fits....but his tone is ‘what if I would’ve made that putt’....
First of all, break a leg on June 15th and let us know how it goes!
How does your work line up with what Dan Pink reports in The Power Of Regret? It's on my biz reading list but haven't gotten to it yet. I'm guessing that you're taking a more systematic approach. Eager to hear more about your research.
This sounds so exciting! One of my unscheduled-future-ideas is to write a "historical" fiction book but with a different timeline. It's not a scholarly idea, but I'd love to learn more about thinking about all the ramifications an unintended side effects should one major historical event have a different outcome.
Hi Annie,
It is funny that you are writing about this. I just completed an assignment yesterday for a Wharton executive education class I am taking with Dr. Platt on neuroscience about counterfactual forecasting.
Best regards,
Chad
Good luck on your defense, Annie! (I’m 61 yo and thinking about pursuing a PhD on my thesis for ethical decision making.) I’ve read Adam Phillips’ book, Missing Out: In praise of the unlived life. It’s a sort of “what would have happened had I done/not done what I did (made/not made that decision)?” kind of book. Have you read it?
In my former professional life, I made the ethically valid decision to say “stop” and my world went tilt as a result. However, no patients were harmed, my coworkers weren’t compelled to violate regulations, and I became a vilified outcast.
Given a second chance, would I have said “stop”? Yes, but I wouldn’t recommend someone else do it. Life presents us numerous dilemmas and the decisions we make define us, our integrity, and the friends we have and no longer have.
Any other good references or words of encouragement? My questioned “behavior” has led me to read many of the popular behavioral economics best sellers. --Gregory
Sounds like the basis for the novel, PostWatch: The Redemption of Christopher Columbus by Orson Scott Card. Basic plot is a researchers who use a technology to perform counterfactual forecasting in the past for interventions from the future!
Judea Pearl puts counterfactual thinking on the third (and highest) rung of his Ladder of Causation. In The Science of Why, he sets these out as
1. Observation, to establish correlation
2. Intervention, to test a causal model by doing something to a key component of the system under observation
3. Counterfactuals, to imagine how the system would produce different outcomes under different interventions that, for practical reasons, cannot be actually performed (the event has already happened, but what if things had been different?) or cannot all be performed at once (should we raise taxes or lower them? all taxes or some? which taxes or how much?)
Agile software development approaches usually include holding a Retrospective: after something is delivered, the team talks about what went well and what didn't, what could be improved etc. An Agile talk recently introduced me to the new idea of a Pre-spective: a meeting before the work starts. Imagine the delivery has just happened and absolutely everything had gone wrong with it. Now exactly what went wrong? What could have been done to prevent it or manage the risk? Who will take ownership of each thing identified?
I've little doubt that clicks and social media attention is not what you are seeking with your dissertation, but can I suggest a name change to "sliding doors forecasting" if I'm wrong.
Good luck today
Congratulations and good luck with the defense Annie!
A strategy professor of mine at Wharton once posed a question to our class: How can our organization know if we're at an inflection point? Given that so many forecasts rely on the future resembling the past, getting the answer to this question wrong has profound implications on organizational failure. It can be hard to see in the moment - signals may be very conflicting. (He used the Kodak story as a counterfactual, and unveiled a significant amount of detail around why the case *for* digital cameras was not nearly as clear cut as people often think it was; the exercise was meant to teach us that the signals were conflicting and Kodak management was not sure if it was really at an inflection point)
I've still yet to find a solid methodology for addressing the inflection question (other than capital available to pursue strategic options), but would be curious if you have any wiser thoughts!
How did you train the subjects to become better at (your words) "counter-factual forecasting"? You'd think that knowledge of the Crimea situation, would assist in making any what-if predictions you might want.
It's a lot like asking "what would've happened had Khrushchev launch the Cuban nukes in Oct 1962?". Easy, we'd all die. Of course if you didn't know that nukes kill people, you wouldn't have come to that conclusion.
In other words, all predictions, contemporary or retroactive, are -- for all practical purposes -- counterfactual. What you don't know, you don't know.
I suppose you might conclude that people with more knowledge are better at making accurate predictions about a certain scenario than those with less knowledge on the certain scenario. But we knew that without the results of these studies.
Interesting article on how Spotify is using Counterfactual Machine Learning Models: https://www.technologyreview.com/2023/04/04/1070885/complex-math-counterfactuals-spotify-predict-finance-healthcare/
Your post reminds me of the Black Mirror movie Bandersnatch. It's about an 80s coder who's adapting a fantasy novel (Bandersnatch) into a video game. It has five possible endings, each based on the choices the viewer made during the interactive movie-watching. While this is new for movies, this isn't new at all for video games. In a way, counterfactual forecasting is at the heart of video-game creation. In your research, did you find any tools that help us do counterfactual forecasting?
Separately, a topic on my mind is how to learn and teach decision-making without bearing the real-life consequences and without waiting for real-life situations to test the decision muscle. That's also partly where my earlier question came from. All the best for the defense, Annie!
I can see the practical value of the counterfactual forecasts for two way door decisions-those where you revisit a decision, perhaps changing your decision. Are there other practical uses I’m missing Annie?
Hi Annie,
This reminded me of when I first read of the term financial aftcasting which I think was coined by Jim Otar. Uses all of the actual market history to give you all possibilities including luck outcomes as my feeble brain understands it.
Best wishes on your upcoming dissertation Annie!
Predictive models assume that everyone (all causal factors) will keep behaving the same way in the future (such as player strategy); making correlation patterns constant. If that doesn't happen, wouldn't that mean the "base rate" could drift off far enough to invalidate a model?
Or what if your base rate is derived by a wrong time scale -- like global climate models using historic (or cherry-picked) versus geologic scales?
Should selection of a base rate explicitly identify the causal factors affecting/driving it?
And should you use a base rate if you can't explain all factors contributing to it?
Good luck on your defense!
Some what-if films...
Groundhog Day
Sliding Doors
Back to the Future
Run Lola Run
Hi Annie. Can’t wait to read it. The burning question was whether your counterfactual training on core concepts were transferable? Did skills, techniques and lessons in decoding a bot for Game x transfer to decoding a bot for Game y? Any insight you can share before the June 15th unveiling?
Interesting topic. I recently came across the application of constructor theory in organisational transformation. Rather than trying to identify cause/effect through reducing complex systems to a component level, it seeks to create counterfactual landscape in order to identify what is not possible. The idea is to then identify "constructors" that will produce replicable outcomes (think machines and culture). https://www.constructortheory.org/
Concepts of a multiple universe -- a parallel world or multiple universes -- have been proposed by some physicists in an attempt to make hypothetical sense of Einsteinian and quantum concepts and functions. While for now unknown (and perhaps ultimately unknowable), carrying around in one's head the concept of an Earth 2.0 is useful in trying to understand Earth 1.0. Much of what occurs here on Earth 1 is the result of aleatory events, random sequences that determine an outcome. Because of our implicit hindsight bias, it is all too easy for us to ascribe purposeful cause and resulting effect to specific events turning out one way or another. Imaging what might have happened on Earth 2 provides a useful lens in helping to recognize that things are less deterministic than they seem. It also can act as a filter in our thought process, enabling one to better recognize the larger, structural elements that create the framework in which random events can create such outcomes.
Hello Annie:
I like to apply what if scenarios to stock market movement. It seems a lot of the moviement is counter intutive. I wonder if there is intelligence behind that, or is it just random? Could a gambler be successful as a trader, or should we all just buy an index and move on?
There is also the aspect of different time frames and sectors. Some say it is a random walk, but others like Buffet make billions.
I would lov your perspective on that. Hindsight is so clear, but the future is so cloudy.
Thanks,
Howard
Most interesting!
Looking back has enormous value for strategic thinkers.
What if's could be potent, if played for example, before a new product launch, they might show a flaw in the roll out plan and one that could be fixed, in effect the team is performing post thinking in real time and one where they can engineer the future. Maybe what if thinking could bring about a new job title for innovative companies, Chief Post Thinking Officer.
This sounds like MLB teams saying “the games in September count the most” when all 162 matter equally.
About your topic, is it a human trait to focus on “what ifs” even if it is the best possible outcome? Is this more phycology rather that “mathematics” based approach?
My son missed being all conference golf team by one stroke. And he’s focused on one putt out of 83 shots. Trying to get him to focus on the totality of the round and want to have a better dad talk...not sure it completely fits....but his tone is ‘what if I would’ve made that putt’....
First of all, break a leg on June 15th and let us know how it goes!
How does your work line up with what Dan Pink reports in The Power Of Regret? It's on my biz reading list but haven't gotten to it yet. I'm guessing that you're taking a more systematic approach. Eager to hear more about your research.