Mailbag!

Photo by Joel Moysuh on Unsplash

What have been your biggest challenges in showing the value of better decision making to people? -Satyajit Rout

The biggest challenge is not in showing the value of better decision making to people since I think everyone knows that better decision making is a worthwhile goal. The challenge is in getting people to do the things that would actually improve their decision making.

A few examples:

1. Creating checklists and rubrics is crucial but it is really hard to get people to create them and even more challenging to get people to actually use them.

2. One of the best things you can do to improve the quality of group decisions is to get feedback from the group independently and asynchronously. In other words, make sure that no one in the group is exposed to the opinions of anyone else in the group before they offer their point of view. The issue is that people love to talk about things in group settings. It feels creative! It makes people feel like they are on the same page and really gelling! Getting folks’ judgments in group setting creates groupthink, cross-contamination, influence, and coercion that result in a group that is less creative and less accurate. But it is really hard to get people to do pre-work.

3. People love to believe in their gut. Getting folks to make explicit why their gut is telling them a particular decision is the right way to go is hard. If you make the rationale explicit, then it can be examined. It is exposed. People might disagree with you. It might turn out your gut is wrong. But that is why making the implicit explicit is so important because that is the only way you can catch an error before acting on it. The reality is that if you want to hone your instincts then you must make explicit what your gut is telling you so you can figure out when it is leading you down the right path and when it is leading you astray.

In your recent work, you allude to coming up with a rubric for making decisions. I have adopted the agree/disagree + reason type questionnaire for gathering information for a decision, and like the idea of some rubric as a way to further remove bias from decisions, but how do you go about creating a rubric for a given situation? Maybe an example would be a client health score or hiring a new person for a role that we hire for regularly like an account manager? -Chris Neumann

Great question! There are a few things with setting up a rubric to know. First, make sure that you are setting it up so answers can be precise and also have enough spread to see dispersion. So, as an example, if you are having folks give a judgment on the statement, “This candidate will be a strong executive partner,” ask them to rate on a scale of 1 to 7 how much they agree with the statement where 1 is strong disagreement and 7 is strong agreement. 1 to 7 allows you to see the spread of opinions so you can see where people on a team disagree with one another. Diving into the dispersion of opinion is where all the upleveling of decisions occurs.

When it comes to forecasts, similarly make sure there is precision. If you are forecasting a sales target, have a point forecast along with a lower bound and an upper bound (this allows people to express both their best guess and their uncertainty about that guess). Same for project timelines, budgets, etc.

That covers some of the form of the rubric, but creating the rubric itself would be an iterative process where the first step is to send out for somewhat free form writing from respondents and then that feedback gets honed into what really matters for the decisions.

Let’s say you are hiring a CFO. You might send out the following to the team to answer independently and asynchronously:

1. Please write a job description for this position. (This allows you to see what each team member thinks the job is. They won’t completely agree!)

2. Please describe to a recruiter the qualities and experience you expect to see in the candidate.

3. What are the short term opportunities this person should execute on or can create within the org?

4. What are the long term opportunities this person should execute on or can create within the org?

5. What are the short term challenges this person will face and need to address on within the org?

6. What are the long term opportunities this person will face and need to address on within the org?

7. Any red flags, disqualifiers you can think of?

After you get that feedback, you can start to see broad themes that you can pull out for the group to rate after interviewing. Maybe categories like: Strong executive partner, Effective team leader, Expert in fundraising, etc. These categories then would form the foundation of your rubric. Once you have a draft rubric you can send it out for comment.

There is so much more to say. I am happy to address it more deeply in one of my AMAs!

Based on your experiences, which bias, or biases, are the hardest for an individual to overcome? Additionally, do you notice bias tends to follow a pattern based on age groups? Lastly, how would you recommend a professional regularly audit their biases? -Derek

All cognitive biases are really hard for an individual to overcome because this is just the way our minds are wired. You can know about overconfidence and still be overconfident. You can know about the sunk cost fallacy and instinctively you will take into account what you have already spent on something in deciding whether to spend more. These things are like visual illusions. I can explain a visual illusion to you but you can’t unsee it. It is the same for cognitive bias.

In terms of auditing bias, the same strategies that help you overcome bias help you audit them. Create decision rubrics, record your judgments, make explicit forecasts, etc. That will create a written record of your decision that you can then close the feedback loop on as the world either unfolds as you expected it to or doesn’t. This will help you see where your forecasts are accurate or inaccurate, where your judgments are spot on or off base, or where the way you were thinking about the decision was biased. And that will also help you improve in your decision making.

New subscriber. Just want to say thank you and ask best way to engage to get most out of membership. -Mark A. Coughlin

Thanks, Mark! In terms of getting the most out of membership, look out for discussion threads that I open up. Most of these will be for paid subscribers only but I do open threads as well. We just did a fun one last week that folks are still commenting on!

Also, I will be doing four zoom AMAs every year for paid subscribers. The first one is set for February 20th at 3pm eastern. Keep an eye out for the link I will be sending out that day.

Other than that, I hope you enjoy the posts and continue to engage in comments and mailbags. I love the interaction!

If you’ve been paying enough attention to the GPT-3 developments, I’d love to know your take on it generally. -Ernie Svenson

I don’t like to comment much on things I am not an expert on so I have convinced Gary Marcus to write a guest post on his thoughts on GPT-3. Look out for that in a few days. Gary is an incredible cognitive scientist and expert on AGI and I have known him since I was in graduate school! I highly recommend his Substack, The Road to AI We Can Trust.