#215 Annie Duke – Episode Transcript

Author. Speaker. Decision Strategist. Annie Duke is one of the world leaders in the decision-making space. As a former professional poker player, Annie won more than $4 million in tournament poker before retiring from the game in 2012.

Prior to that, Annie was awarded a National Science Foundation Fellowship to study Cognitive Psychology at the University of Pennsylvania. Annie is the co-founder of The Alliance for Decision Education.

Her first book Thinking In Bets is one of Sean’s favorite reads and her latest book How to Decide: Simple Tools for Making Better Choices is the new playbook for making better decisions.

Get ready to improve your decision making on this episode!

Transcript

Sean Delaney: Annie Duke, welcome back to What Got You There. How are you doing today?

Annie Duke: It’s a lovely day. sunny out and I’m doing well, how are you?

Sean: I’m doing fantastic. This is one of those conversations that fortunately get to have once a year now with you, and I love it. I walk away with more takeaways, and you were first on episode 132, and we covered more of your back story and your first book Thinking in Bets.

So we’ve got a lot to catch on since then, but I’m wondering, is there any skills training that you do every single day? We can kind of think of these mental calisthenics or things you would do on a daily basis just to improve your knowledge. 

Annie: Oh, that’s a really interesting question. To improve my personal knowledge, in other words, not what I’m doing with my clients and whatnot.

Sean: That’s correct. Yep.

Annie: Yeah, okay. So one of the things that I really try to do is two things. One is make sure that trying to live sort of the principles of the type of decision-making that I’m trying to spread throughout the world. So a little bit of the structure in the way that I try to think, so that can manifest in a whole bunch of different ways. I love to play tennis, and I take it very seriously, and so I really try to practice humility about what do I know? What do I think I know? What don’t I think I know? How willing am I to go backwards and undo things in order to test them out and experiment with different grips that might… Grip changes or stroke changes. The willingness to bear all of the mistakes that you make, thinking about what the longer term goal is, and that’s actually hard. It’s like when you change, if you change your forehand grip, you all start… Depending on which way you go, if you’re right-handed and you bring your grip to the right, you’ll start hitting balls into the bottom of the net. Ultimately the goal is that your forehand is gonna be better, but you have to bear a lot of short-term pain in order to do that, and you have to be willing to see the point of view of the Pro that’s working with you, as opposed to your own point of view in order to be able to do that. And I feel like that’s a place where I got to actually practice this stuff a lot, and kind of an unusual… Obviously, the tennis court isn’t necessarily where people think that I would be practicing that. 

The other thing is that I do actually really try to make sure that I’m exploring a lot of the kind of info… I’m sorry, I really try to make sure I’m exploring a lot of the information ecosystem in order to make sure I’m getting a pretty broad swath of the, what you would consider the reasonable views of what’s going on in the world, and sometimes it’s actually really good to understand the unreasonable views and be challenged by those as well, ’cause sometimes what you think is an unreasonable view has some nuggets in it that maybe are things that make you think that actually turned out to have some reason to them. 

So I think I’m a pretty broad consumer of news, and then I have a relatively broad spectrum of where my friends sit as well in terms of the type of conversations that I’m having. That’s outside of the work that I do, where I’m really trying to help clients think through problems in which the answers are gonna be subjective, where you’re talking about subjective judgments, where I’m having to just exercise those muscles all the time and learn, that’s where I learn a lot of things about my own work in terms of how resistant are people to these types of ideas or how open-minded are they to them, where are the places where there are hiccups, what are the things that are reasonable to expect somebody to be able to implement in their own life, how long does it take? What are the different solutions that I could be exploring? Not trying, really not to settle into the idea that I might know the answer, but exploring really trying to be aggressive and exploring other ways to improve upon the process, even if the process is going pretty well. So I think they get a lot of opportunities to actually be kind of instantiating this in my daily life.

Sean: No, I think that answer just really culminated all of your approaches and that cross-disciplinary approach, and why I think it’s so helpful for me in my own life, and I know a lot of the people I’m in communication with, your work, because you’re getting so many different inputs. I would love to hit on what you were discussing there for a second about tennis, and when you’re looking to improve on a skill, do you start with an end goal in mind, or what is that beginning approach like for you?

Annie: So I think that we can have finite goals. I don’t know if people are familiar with the tennis rating system, but you could be like a 3.0 or 3.5 or 4.0. As you get better, your rating goes up and a Pro, they don’t really have a USTA rating, but they would be above a 6.0. So Pros are sort of off the chart, but a really, really good, like the best amateurs out there are probably 5.5. So your average… I would say the average really solid amateur is like a 3.5. So I could set a goal for myself that I wanna be a 4.5, but I think that that’s actually not so useful. For one thing, I think I can create perverse incentives ’cause there’s ways to game the rating system without actually… You obviously, to go from say a 3.5 to a 4.5, your skills would have to get better, but you wouldn’t necessarily have to be the best expression of a 4.5 in order to get to a 4.5 rating, because there’s ways that you can actually game that system, and I guess maybe because I’m a poker player, I sort of think about what are the holes in the system that can allow you to move up more quickly than maybe your skill would allow. So I actually don’t like to focus on that, what I’d like to focus on instead is goals that have to do with process. So for example, I would like to, ultimately I would like to create more spin on my forehand. Now, what the end point of that is, is unclear because I can always be fooling around with that and figuring out what type of spin, when to spin, and I’m trying to learn all the different spins, but it’s not like I feel like I’m ever finished. And I think that for a lot of the things that we do in life, it’s good to feel like there isn’t a finish point, but that you do have solid things that you’re trying to achieve, nonetheless. It’s just that they’re sort of continuous in nature, that there’s always something to build upon that.

And that’s absolutely the way that I set my goals on the court.

Sean: Gotcha, yeah, no, I love the infinite approach there, I think it’s incredibly helpful to approach life that way. I’m always intrigued with high performers, how do you solve for that balancing act between focusing on your strengths and doubling down on those as opposed to going after your weaknesses?

Annie: Yeah, so that’s an interesting question. Actually, I think that this relates… I think this relates to my book. So I have a section in the new book How to Decide, it’s Chapter 7, which actually addresses analysis paralysis, but what I’m trying to do is get people to sort of think about what is the situation that I’m in, what type of decision am I facing so that I can understand sort of two things. One is, what’s the impact going to be if things don’t work out well, and then secondly, what is my optionality, like how easy is it for me if I choose a particular path, for me to then reverse course and get back to an option that I might have rejected or maybe choose a brand new path that happens to have opened up for me? So we can think about impacts and optionality.

And basically, what I talk about in that section is that the lower the impact of having a bad outcome come from the decision that you make, the faster you can go, the more experimental you can get in your choices, and likewise, the more optionality there is the easier it is, I call it quit-tuitevness. So people talk about sticktoitiveness, you gotta stick to stuff in order to succeed. I say, well, actually quitting a lot is what allows you to figure out what to stick to, and that’s ultimately gonna make you succeed more because as you’re sort of poking, I’m trying something and then it’s not working out, I’m quitting, I’m trying something that’s not working out, I’m quitting. I’m gathering all this information.

So I think about that the same in terms of when are you focusing on your strengths versus your weakness. Well, when you’re focusing on your weaknesses, what you understand is that the likelihood that you have poor outcomes that come out of that are just kind of higher. So depending on the situation you’re in, if you’re in a situation where having a poor outcome doesn’t have really high impact, you should try to get really experimental and try to really start focusing in on how do I actually think about how to improve my weaknesses in a situation where I have a lot of leeway to do so? So if I were to translate to that to the tennis court, since we seem to be talking about that, obviously when I’m practicing with my Pro, I can be focusing quite a bit on my weaknesses. I’m training my strengths as well, but I’m thinking about different weaknesses that I really wanna be focusing on during some time period, and I’m focusing on those in a situation where if I hit it into the net or I hit it out of the court, it literally doesn’t matter.

Another place where I can think about really trying to increase the quality of the things that I’m weak at, is if I’m playing a player who’s much worse than I am, because if I’m playing somebody who’s much worse than I am, I have more leeway to do things that aren’t so good, because my edge is so great that it’s very likely I’m gonna win anyway, in other words, the things that I’m weak at are likely to be better than the things that they’re strong at. So that allows me within the context of a match now, to start really working on my weaknesses, but if I’m in a really tight match where hitting a bad shot, where hitting something into the net or choosing to hit the ball into the wrong part of a court or chipping when I should have been hitting a topspin ball or whatever it might be, when that’s really gonna have a big impact, it’s better in those situations for me to be sticking more to my strength because I just have less room for error.

Sean: Interesting, so is it in context where you’re less experienced, it’s better to take those chances because the feedback loops are quicker and clearer, and then you can build those up and improve on them, and then when the states really matter implement your strengths more?

Annie: Yeah, that’s what I would say. And by the way, when you get to that point, you’re gonna have more strength for having worked on the things that you were weak at. Whether it’s weaknesses or strengths, the thing is, it could just be how fast are you deciding, because when you’re thinking about decisions, there’s a trade-off between time and accuracy.

So generally, not always, but generally the more time that I take on a decision, the more knowledge I can gather and that knowledge should be creating more certainty for me about what the possible ways that any option that I choose could unfold, and that means that my decisions in general are gonna have more accuracy in general. There’s a relationship between time and accuracy. The thing is, though, that time is a really valuable resource. We have a limited amount of it, and we want to figure out where we should really be putting our decision-making time and where it does not kind of matter. Strength and weaknesses sort of fit into this category, but so does… I don’t really care if you’re super strong at data analysis, if the impact of the decision is not gonna be very great, and particularly if the decision is reversible in some way, I would not spend a whole lot of time really analyzing the data. I would just do whatever it is, because in the doing, in making the decisions, you’re actually allowing the world to give you feedback and tell you whether it’s working or not, and this then creates a really tight and fast feedback loop, and it’s kind of the idea behind if people are familiar with Agile software development. It’s sort of the Agile movement in businesses. I’m gonna release a feature, I’m gonna do it pretty fast, because I’ve thought about this particular change that I’m making to the… Let’s say the user interface, and I realize that if it fails, my customers aren’t gonna really care and I can roll it back super fast. So I’d rather be doing these small things that are a little bit low impact, lower impact and easier to reverse in order to collect information about what are my users like, which I may not be able to get from data because I may just need… At some point, I just need to ask them, right, what works? If I take certain shortcuts in coding, if I try to write the code in a different way, does it work, does it not work? And you’re constantly trying to look for these places where you can shave time off, and then what that allows you to do is gather information and build much better models of the world such that when you do face a really big decision, you’re gonna have much more information and it’s gonna be a better decision for it for having done all these little things first. 

Sean: No, that makes a lot of sense and has a lot of clarity to that. You were talking about analysis paralysis a second ago, I thought you had some very clear examples in the book that a lot of us run into every day. Any of those real life examples, I know around eating it and selecting food choices, that you think are just incredibly helpful?

Annie: Oh, well, my gosh, I mean, do we not all know… Or maybe it’s you, the person who spends 15 minutes trying to order off a menu. Obviously this is in the before times, before COVID, but we all know those people and we may be those people ourselves. It’s just very common that you go into a restaurant and it’s just you’re trying to decide between two entrees and you’re quizzing everybody that you’re at the table with, and you’re calling over the wait staff and asking what dish do they think is better, and you’re looking at Yelp reviews and all this stuff, trying to figure out what am I gonna order? And then it happens the same thing with choosing something to watch on television, people will really agonize over those decisions and have super long discussions about them or what to wear, and if you think about it… Particularly if you take something like what to watch on television, if you don’t like it, you can shut it off and choose something else, it’s just like, why are you spending 15 minutes having an argument about which show to watch? It’s not a permanent decision, but the problem is that we really look at it as if it is. Like once we watch the show, I guess I’m gonna be watching the show for the rest of the night, and I’ve got no option if I don’t like it to do anything else, which when I say it out loud, sounds so incredibly absurd, but I think that that’s the way that people approach it, like I’m trying to decide between watching The Crown or Tiger King, and somehow it feels like the most important decision that’s in front of you and the family is having a big argument about it, and it’s like, Okay, but if you don’t like Tiger King, you can just shut it off and go put on The Crown. So we don’t really think about these things that way, and I think that the statistics show that if all you do is take how much time do people take ordering on average, how much time do people take like picking the outfit they’re gonna wear for the day, how much time do people take choosing what to watch on Netflix, I think it adds up to something like almost seven business weeks of time in a year that you’re spending on what? When you step back from them, it seems so clearly to be decisions of very little consequence, and yet we’re using up nearly two months of six weeks of our lives every year on those decisions.

In that particular chapter, I’m really trying to give people a path to how do you think about your decisions such that you can get out of this problem and understand when can you go fast and when can you go slow? And the answer, by the way, which I hope you got from that chapter is mostly you can go fast, and then very occasionally you wanna slow down and really take your time, but mostly you can be going fast, and I think that part of the problem is that we kind of revert a lot of decisions that you could be going super fast on, we’re going to really slow on, and then a lot of the decisions that you should be stepping back and taking your time with to kind of work through them because they are gonna be high-impact, people are just going with their gut, and so it’s like they’ve sort of flip the script in a weird way and they’re actually allocating their time almost opposite to the way that it would make sense to allocate their time to those decisions.

Sean: Yeah, I thought that chapter, for me, added a lot of clarity, because there are certain things like what I wear, I don’t even think about it. I kind of have just the same outfits, but some of those other decisions, even around food choices, I was realizing I was spending and allocating so much of my energy and effort towards those, and then when those high impact decisions came up, I was already drained and exhausted from some of those little ones. So I thought that chapter just added a ton of clarity, and believe me that those statistics that you had included in there were fascinating. I’m certainly highlighting that chapter, sending it to some friends who need it.

Annie: Oh, thanks.

Sean: But I would love to even just hit on how you start off the book, and you’re talking about two things that basically determine how your life turns out, and that’s luck and in the quality of your decisions. So I think having a book based on how you make decisions is incredibly important, but when making important decisions, what are, or should the objectives be? How do you approach that?

Annie: Oh gosh, well, you should buy my book.

So I think that what I’d love to sort of lay the foundation for this, what we can think about is that when you say there’s two things that determine how your life turns out, right, there’s luck, and there’s the quality of your decisions, and the thing is luck, you can’t do anything about. Luck is something that you, by definition, don’t have control over, so I wrote the book to get a focus on the quality of your decisions, but within the quality of your decisions… Here’s the thing that I think is really important to understand is that the foundation of every single decision that you ever make are the things that you know. So we can think about our own knowledge, what are the things that we know? And if we were to draw it out, it’s like a speck of dust that fits on the head of a pen, and the stuff we don’t know is the size of the universe, and this is really the main thing outside of luck that really kind of mucks up our decisions. So you could think about if I could remove luck from the process and I could remove this information problem that our information, whenever we’re making a decision, we don’t have all the information that we need in order to make a perfect one, then all that would be left would be sort of like, how are you analyzing that information?

So it would be like you had a crystal ball, the crystal ball showed you the future, and then there’s some stuff to clean up because you may see a future where you get in a car accident and still choose it for some reason. So we’d like to get you… If you could see the future, we’d like to help you to figure out what makes something a great choice, but before you ever get to that, you have to sort of think about this information problem, and how are you cleaning up the information? So what I’m trying… What I’m really trying to communicate through this book is that unlike luck, you have some control over what you know and what the quality of your information is, and if you can think about, there’s this whole universe of stuff that I don’t know if I could actually be peering into that universe, taking, looking into it and then walking through it in a more random way, such that I was colliding with all sorts of new information, such that I was colliding with information that actually corrected the inaccuracies and the things that I believe, and I was approaching that information in an open-minded way, this would go a long way to really helping the foundation of every decision that you make, which is ultimately informed by the things that you know, because if we think about that foundation that’s at the base of all of our decisions, there’s two problems with it. We have two issues. 

The first is that there are cracks in the foundation, and that’s that we believe things that aren’t 100% true. They’re probably not 100% false either, they’re somewhere in the middle. But those are cracks. Our foundation is cracked because we have inaccuracies in the things that we believe, and then the second problem is that the foundation is flimsy. It’s not broad enough because there’s so much that we don’t know in comparison to the things that we do, and the solution to both of those things, repairing the cracks in your foundation and also broaden your knowledge is to get a really good look into that universe of stuff that we don’t know, and so that we can find all that information that’s gonna help this foundational problem. The problem is that we don’t do that.

We’re pretty bad at that piece, and so what I hope this book is, is really an argument for not just philosophically why you should be really trying to be eager in increasing your knowledge and the most open-minded way possible, but also how you might do that.

Sean: I absolutely love that. And even how this conversation started out and you taking on information from a lot of different people, a lot of different sources, and you worded that as colliding. I kinda think of that as that the cross-pollination of ideas from different fields, but the big thing there, and this continues to get hit on throughout the book is just the openness to new ideas, and it’s just such one of those intriguing, complexing things to me that so many people are unwilling to be open to new ideas and to test what they know and what they don’t know. So yeah, I think that’s one of the big takeaways and one of the very clear things I saw.

I’m wondering, what have you found, because I know you work with a lot of investors, a lot of people across all different industries, what has been one of the elements of decision-making that’s just harder to teach?

Annie: That’s interesting. I would say… So there’s two candidates, so I’m gonna give you both candidates. We’ll start with number one. I think the first thing is that it takes a lot of work to help people understand how to think probabilistically, and what do I mean by that? Just for any decision that you make, there’s a set of possible ways that it could turn out, and each of those possible ways has some likelihood of occurring, and it’s incredibly rare that you could make a decision which is gonna go a certain way 100% of the time. As an example, even decisions that we think are like that or near certainties aren’t. So if I’m two feet from a car that’s in front of me, and I slam on the accelerator, it may feel like it’s a certain outcome that I will hit the car that’s in front of me, but we know it’s not certain, because there’s some small chance of mechanical failure of my car as an example.

So even things that feel very, very certain are actually not certain, and when you get into whether a particular sales strategy, like how much is it gonna increase demand if I pick a new advertising channel? Is that gonna lower my customer acquisition cost? So on and so forth.

If I choose a particular diet, how much weight might I lose, or how much muscle mass might I gain, or whatever it is that my goal is within a certain period of time? Now you get into things where there’s… You start to get a pretty wide range of possible outcomes, and we can see this in COVID, so… Wow. When’s the vaccine gonna come? Each vaccine that’s being tried may work, may not work, maybe it’s 30% effective, could be 60% effective, because there’s all these different ways that these things could unfold, and so you think about there’s many different futures that can result from wherever you’re sitting now, and those are all probabilistic in nature. So first of all, getting people to kind of think about how would I try to figure out what the probability of those different outcomes is, and how would I go about actually forecasting that and how willing am I to do that? That’s actually quite hard. And then the other thing that goes along with that first point has to do with this really weird thing about probability, that’s just really hard to get people’s heads around, which is that if I tell you there’s a 30% chance of rain tomorrow, it either rains or it doesn’t. So it doesn’t like 30% rain. So the 30% says that under the conditions under which I’m forecasting, there’s a 30% chance that in the area that I’m forecasting with, it will rain somewhere within my forecast area. I think that that’s how it did. And for weather buffs, if I said it wrong, I apologize, but I think that that’s what it’s saying. So what that means is that if I were to… The way that I could judge you as a weather forecaster would be to look at a large data sample, like a large set of forecasts that you’ve made for that coverage area and find out when you say that there’s a 30% chance of rain, does it turn out that it rains in that area 30% of the time, somewhere in that area, 30% of the time, but I can only do that over time, and the problem is that the next day when I’ve told you there’s a 30% chance of rain and it actually rains, because it’s either gonna rain or it’s not, you now either say, well, it rained, she was wrong because she said there was only a 30% chance. And look, it’s pouring. Or if it doesn’t rain, you might say, I’m right because, well, I said it was a 30% chance of rain, so now maybe I’m right that it didn’t actually rain, but I’m neither right nor wrong.

And I think that’s really, really, really hard for people. You can see it with elections, it’s like Trump is 35% to win or 30%, whatever it was, to win the 2016 election and he won, and everybody’s wrong, except that was one try.

So I think that this kind of being able to live with probabilities as opposed to yes or no answers, I think that’s incredibly hard.

And then the other piece, the other thing that I would say is really, really hard, is to get people to talk to each other in a way where they’re not trying to convince each other of anything, they’re just trying to convey the information that they have. They’re just trying to help. They’re just trying to say, here’s my rationale, this is why I believe this thing. Without trying to get you to agree with me, and that’s actually the second thing that I think is probably the most challenging.

Sean: For both thinking and living with probabilities and then not trying to convince someone necessarily, is there anything you’ve seen people who weren’t good at that initially, but be able to develop that mindset? Is there anything they do, or is it just one of those personality types that they’re open to changing?

Annie: Yeah, so obviously, personality type really helps, but I think that particularly when you think about the fact that you have influence over the people that are around you, and this is particularly true if you’re in a leadership role, that you can really create a team dynamic that allows for this thinking, this way of thinking to really bloom. What I talk about at the end of my book in particular, is how are people in groups interacting with each other such that you can allow all the dispersion of opinion. All this cognitive dispersion, all the different perspectives that are living on that team, to feel like they can breathe, and then allowing that to live, to not feel like the end point of the conversation of the team is for everybody to agree with each other.

So how do you do that? Well, you have to show people that you’re actually interested in getting their opinion, you don’t… That you’re not just looking for consensus, that you actually want to be informed by their opinion, and you need to do it in a way that allows them to not feel like they’re disagreeing with somebody.

So let me try to explain what I mean by that. Let’s say that I was asking you what your opinion on Forest Gump was, whether you liked it, whether you thought it was a good movie or not, and… So I’m asking for your opinion, but we know that human beings don’t particularly like to disagree with other people, so if you’re gonna express your opinion to me, what is the thing… What is the thing that you would need to know from me first, to realize that your opinion might disagree with my opinion? What is the thing that I must tell you for you to know that?

Sean: What your opinion is?

Annie: That’s right.

So this seems so simple, right? It’s so obvious for you to just… For you to know you’re disagreeing with me, I would have to tell you what my opinion is first. So let’s think about the way that normal human interaction goes, this is whether you’re in a leadership position, on a team, whatever. I will say to you, I thought that Forest Gump was really overrated as a movie. What do you think? Sound familiar?

Sean: Too often that happens.

Annie: Right? So the problem is that now, if you thought it was the greatest movie of all time, and particularly if we’re on the same team where it doesn’t feel good for us to sort of… We sort of interpret being on the same page is like consensus, this is kind of how we get to group things, you’re probably now gonna not tell me your true opinion.

It could be that, for example, I’m a famous movie critic in particular, which is sort of the same thing as being a leader on a team, right? So if I’m the person with the power here and what’s considered the higher quality opinion in general, for whatever reason, because I’m like the most famous movie critic in the world, a few things can happen. One is, you may not express your opinion just because you don’t wanna look like an idiot because, well, this fancy movie critic thinks it’s overrated, so I’m just like joe schmo  over here. I don’t wanna say I didn’t really… That I loved it.

So you might not tell me for that reason. You might also not tell me because maybe we’re in a group of people, you don’t want to embarrass me, because maybe you think that I’m just wrong, but you don’t say so because you don’t wanna embarrass me, and then the other thing that can happen, and this is particularly true if you’re speaking to someone who’s… If you’re in a leadership role, is that Sean, your opinion might bend toward mine without you even knowing it. So it could be that if I could have gotten your opinion without telling you what I believed, you would have said I think it’s really overrated, or you would have said that you thought it was really great and it really deserved the Oscar, but if I tell you my opinion first, you without knowing it may start to alter your opinion toward mine. So you might say, for example, so somehow your opinion might change well, at the time, it seemed like a really amazing movie, but it hasn’t aged well. It could end up being something like that, which is not actually what your opinion was, but you’re unconsciously starting to conform your opinion to mind.

So what happens when we end up having all these conversations in a group is that there’s a lot of value placed on agreement. The group tends to linger over the places where they agree, they tend to think they agree more than they do, and sort of declarations of yes or no, getting to consensus becomes what’s valued, but if what I do instead is I figure out a way within a group setting to not let people know what everybody else thinks before they express their opinion, which I can do, then I’m now getting those people to understand that I actually care about the differences and less about the consensus, and that allows people to understand that it’s not all about agreement, because through this kind of systems approach, you start to allow the dispersion of opinion to thrive and breathe, and all I have to do is this simple thing, which is ask everybody independently, not in a group setting, before you come in and start talking about it in a group setting. It’s like the simplest little track, it’s like magic.

And then what I’ve seen happen with people is that even people who let you come into it naturally, less open-minded, start to really get into it, because it starts to sort of define the team, and I’m constantly reinforcing this idea that, no, you should express your opinion. I care about it. Because now when I’m looking at it’s so interesting, like Sean, you’re sort of sitting here as an outlier. Could you please just give your rationale for why you think that? And I’m sort of celebrating your difference rather than saying defend yourself, which is a very different way to approach it. So I think it really has to do with what’s the kind of culture that you’re creating, and then what are the systems that you’re putting in place? What are the strategies and the tactics that you’re putting in place in order to actually create this kind of attitude in people? And then you can always move people even the least open mind in person can get moved to bet.

Sean: It’s so simple, but so rarely done. I’m wondering if you’ve seen organizations bring this methodology on? Is there a downside and negative consequences of this that we just haven’t thought through yet?

Annie: Well, so I wouldn’t wanna say no because I don’t know, maybe, but what I’ve seen with the people that I work with is that there’s all… The downside is you have to do pre-work. So you can’t just come into the meeting on the fly, but I particularly think that when people in particular, I think that when people come into meetings on the fly, the meetings tend to be not so great, but you’re definitely gonna have to do pre-work, so you’re gonna have to ask… You have to ask yourself, What’s the feedback that I’m trying to get from the group in terms of if we’re coming into discussing something? So as an example, let’s say that we had a hiring committee that had four people on it, we have to figure out what are the opinions that I’m trying to elicit from each of the four people on the hiring committee about the candidate under consideration.

So those might be things like a forecast of what’s the probability that candidate will still be here in six months, a year, two years? How good of a cultural fit might they be? Let’s say they’re in a role that’s outward-facing to customers, where do they sit on a scale of zero to five in terms of how… What I predict their relationships with customers are gonna look like? How strong those relationships will be, so you can see… I’m just throwing these things out there, but you figure out for the job that you’re hiring for, what are the qualities, what are the things that you’re implicitly predicting about the person under consideration that, in that hiring process, make those explicit and have people actually write down some sort of rating or forecast of those things with a bit of a rationale as to why they think those things. They have to do that in advance. So we have to think about the pre-work now. You have to figure out what are the things that I’m gonna rate them on, and then people actually have to fill those out with a little bit of a rationale, and then each person does that independently, and then everybody kind of reads that in an anonymized fashion, so that’s work.

So that’s really what I’ve seen is the main downside, but other than that, I would say that the upside is that it creates accountability to the decisions that you’re making, to the things that you’re forecasting when you make decisions, so that you can actually get a very clear look back, because a lot of the problems that we have with learning is that we miss remember the things that we knew at the time of the decision, so that makes it really, really hard to learn from experience, because you can’t really close that feedback loop with any fidelity, because you just don’t remember what you know. So the pre-work sort of naturally creates a record for yourself that you can actually look back. So I think that that’s really, really helpful. 

The accountability itself and actually having people do forecasts makes them do the thing that we talked about before, which is, okay, I know the things that I know, but now I actually have to write down what I think that this probability is or where this person sits compared to other candidates that I might see. Maybe I should go look stuff up. So it gets you to actually start to think about what are the things that I know and don’t know, and what could I find to make my forecasts more accurate that I have to know make, because I have to write it down. And so that gets you really thinking about that universe of stuff you don’t know in a high quality way, so I think it improves knowledge in that way. 

I think here’s a problem. I think that people will say like, Well, I just have a good gut feeling about this job candidate, for example, but in that gut feeling, like I said, there’s always things that are implicit in the decision that you make, even if you’re making it with your gut. So if I’m hiring a salesperson, for example, even if I’m going with my gut implicit in that decision, I think that the person is gonna be very strong in sales. Let’s just say this very simply. Right. So I think that the more that you can make explicit and have to say out loud and give a rationale for those things, the more that you can make explicit things that are already implicit in the situation, the better the quality of the decisions that will come out of the process, because you’re just surfacing those things and then through this process you’re surfacing the differences in opinion that might might otherwise get hidden by group discussion, which I think is all really good. And then the interesting thing that happens is that when you get into the meetings, the meetings are more efficient, and the reason I think that the meetings are more efficient, and you can tell me if this is your experience with meetings, is that meetings tend to be a lot of lingering over the places where people agree. So you’ll express, I think that this person is gonna be amazing in sales. I think they’re really personable and very charismatic and I think they’re gonna be really strong in their relationships with our customers, and then what I’ll say is I totally agree with Sean, and I would also like to add my opinion into the mix, and I would also like to spend three minutes telling you why I think that person is gonna be very strong in sales, and the majority of the meeting goes on like that. Whereas if you’ve done the pre-work, you can see that… You could already see that Sean and Annie agree on that. But now what you see is that we disagree on the likelihood that the candidate is still gonna be with the company in a year, and we have a rationale for that. So now what you can do is you walk in the meeting and very efficiently stay here. Okay, here’s all the places we agree. Yeah, yes, we all agree the earth is round, great, but let’s start to explore these areas of disagreement. These areas of dispersion that we’ve surfaced, not with the intent that I should convince Sean of my position, or not with the intent that Sean should convince me of his position, but rather so that you get to say your rationale for why you think it’s really likely that that person will be here in a year, and I get to give my rationale of why I think it’s really unlikely that that person will be there in a year, and we’re just trying to inform the group of why we think that. 

Now, we may at the end, I may raise my hand and say, I’d like to revise my opinion because Sean thought of some things that I hadn’t thought of, which I’m really thankful for, but that’s because I listened to you convey to me why you thought that not because you were trying to convince me of anything or tell me that I was wrong. So aside from more work in front, I haven’t really seen much bad come out of it, but let me just say I’m open to… There could be very… There could be bad things that reveal themselves later to me. So I wouldn’t throw aside the idea that maybe I’m biased in the way that I look at it.

Sean: Yeah, I ask the question because this is something I try to do in all the businesses I’m involved with, and you bring up a really interesting thing, and that’s almost that when we slow down and do the pre-work, we actually speed up in the long run. Those meetings do become so much more efficient, so if you come across any potential downsides, please let me know. I would love to know that.

Annie: Well, absolutely, I will absolutely let you know. Look, here’s a downside.

There may be someone on your team that’s like just super resistant to it, and you get in the conversation and they don’t wanna just convey why they believe what they do, and maybe there’s someone where you’re trying to convey your meaning and they keep saying, but wait, you haven’t thought about this, or you’re not thinking about the data, or why haven’t you considered this, which is, they shouldn’t be doing. They’re not supposed to be interrupting you, you just get to convey and then I can ask for clarification, but I’m not supposed to tell you you’re wrong. I’m not supposed to say, well, you haven’t thought of this because I get to convey that. It shouldn’t be me trying to convince you or tell you you’re wrong. There are some people who don’t… Who don’t do well in that kind of environment, I suppose. Maybe aren’t open-minded to it, but… So some people may not wanna do it, and you may lose them as employees. I gotta say, my argument would be, I’m not sure that that’s a bad thing.

Sean: Yeah, completely agree there.

Annie: I mean, if you’re trying to build a culture, right, within your team, that particular type of behavior, I don’t think it’s really helpful to the team, and one of the things that I think is really nice about it is that as we have discussions about diversity and inclusion, I think that we need to keep our eye on the inclusion part.

And so the question is, if someone’s coming to the team who has different experiences with the world than we do, and the world has interacted with them differently than we have, and they may have very different perspectives, we wanna be able to surface those. And we want them to feel like those just live, that they live that it’s okay to have those on the team and that they don’t need to conform to whatever the prevailing opinion is. They don’t need to feel like to be on the same page, I have to have consensus with what everybody thinks, and so I think it allows for much more inclusion of cognitive diversity in a way that really improves your decision process, because you’re so much more informed as a decision maker if you can access all the different points of view of the people on your team.

Sean: Once again, this is why you’re reading is just a must. The frameworks just make everything so much clearer, and one thing I could use for a little more clarity is when we’re making decisions, and they have long feedback loops, so say you’re making an investment and not in the public markets and the private markets, so you might invest in a company, you might not get a return or find out for six more years, so then how do you build in quicker feedback loops when the payoffs way further down the road?

Annie: So that is such a great question.

I mean, really great. Thank you for asking me that. That’s awesome. So what I would say is that that goes back into this idea of what’s implicit in the decision that you’re making. So if you’re investing in a company where the exit might not be till six months from now, think about what are the reasons that you’re investing in that company? There are certain things that are implicit in the forecast about… And many of those things are things that you know will happen in the short term. So as an example, let’s say I was an angel investor, right? So I’m just a founder has come to me and they have a really good product idea, right, and I just think that their product is cool. Implicit in my decision to invest has to do with things that I think that that person is gonna do along the way. They’re gonna hire an amazing team around them that can sell the product. Well, those are things that are gonna have a much shorter time horizon. They’re actually gonna be able to execute on their development timeline, so I can be forecasting that, right? They’re gonna be able to raise the next round of capital after angel, I can be forecasting that. Do you see what I’m saying? So it’s not like you make the decision and then the world is a black box to you until that company exits or it doesn’t. There’s all this… Within that very long time horizon, there are all these things, these sign posts that happen along the way that you can stop that and look at, and when at the time of the decision, you should be actually trying to forecast those, in other words, you should make those explicit, because then you can create much tighter feedback loops.

Sean: No, that’s very helpful there.

You mentioned doing that on the onset and kind of having those goal posts. How about as that progresses, should you constantly be adding new ones and new metrics?

Annie: Well, yeah, so again, such a great question, You actually get into places that hat I didn’t get a chance to cover in the book, so I really appreciate that. That’s a really good question. So the answer is yes.

So you can think about when, if you’re reviewing an investment, like whatever that periodic review that is that you do of the investment at that moment, the investment may be a liquid, but you should do this idea, but let’s just make it easier. Just an easier way to get your head around it, let’s assume that the investment has some liquidity that you could sell your position. When you’re reviewing… If you’re reviewing that investment, you should think about it as a brand new decision. If I had no money in this, would I be investing?

Sean: Such a great point.

Annie: At which point it should just be re-forecasting all of that stuff, and then depending on what those forecasts look like, you could either say, I wanna sell the position or I wanna hold, or maybe you wanna press. You could wanna invest more heavily in it depending on what the result of that forecast looks like. Sort of the Netflix example, and when there’s liquidity and the investment you can think about investing in your time to watch a movie is a highly liquid investment because you can literally turn the TV off after five minutes and stop, so some investments are more like that, you can get out of them faster. So now we can sort of just take that framework and move that into something that is less liquid. So if I have an investment where there isn’t a way for me to easily get out of the position, I still wanna be doing those regular reviews and re-forecasting, and the reason is that one, I’m gonna be making more decisions in just that one, and the more that I do these kinds of forecasts and I allow myself to close those feedback loops more quickly, the more that my decisions are gonna get better because my knowledge of the world is gonna improve. It’s gonna help me correct my process, and I’m gonna build better models, that’s basically what it is. I’m gonna make better decisions going forward because I’m closing the feedback loops more quickly. I’m not just allowing the world to sort of wash over me. And given that I’m gonna make more decisions, I’d like to sort of understand the area that I’m making decisions in a lot better, which that helps you do. 

The other thing is that when you make those forecasts, even for something that’s not liquid, it may turn out that you say, Oh, this is really not forecasting well. I don’t really like this position much, but you can now say, given that that’s not forecasting well, that may reveal other investments that you can make that act like excellent hedges against the one that you’re already in, even if the one you’re in doesn’t have liquidity. So by doing those forecasts, you can open up, you can sort of see other market opportunities that might be available to you and also creative ways and more accurate ways to actually find ways to mitigate the downside of the choice that you’ve made, that it’s harder to get out of.

Sean: Yeah, that last point you bring up the creativity element of all of this, and I found that re-forecasting really does open up so many more creative ways of thinking and approaching those certain problems. This is why I love these conversations because you help spark new ideas in me, but I’m wondering, I mean, you’re doing a good job with helping me learn these things, what about outside of decision-making, what skill of yours or mindset do you just have the hardest time teaching other people?

Annie: That’s an interesting question.

Okay, so let me think about this. So that’s an interesting question, what’s the thing that I have the hardest time teaching to other people… So I think sometimes to a fault, I think I’m just kind of so… I’m thinking so much all the time about what I can control and what I can’t… I think this kind of comes out of having been a poker player, that you just have to so get used to… That card was gonna hit 2% of the time, and I just happened to be here for it kind of thing, you know? So I think there’s… Gosh, and I’m gonna say two things. The first is, when I’m really annoyed by a bad customer service agent, I don’t realy… I’m pretty calm in those situations, because I think that I can sort of step back and I’m always sort of living a little bit farther into the future, I think that then people naturally do. And so I’m imagining if I start yelling at this person, is this actually gonna get what I want accomplished? So that track is going on in my head at the exact same time as a person’s being incredibly annoying and frustrating. So then I’m just saying, Well, no matter how frustrated with the person I am, I’m gonna be really friendly to them because that’s the way I think that I’m actually gonna be more likely to have this turn out well, and I think that that’s actually something that just it was so many years of training, at poker of just like crap happens. You have to think about the end goal and not get caught up in what feels really good in the moment emotionally, because it’s probably not gonna help you get to the place that you wanna get to, and I think that that’s actually kind of hard to teach. 

I think that a lot of that is sort of experience and some of it may just be naturally, I just happened to be someone who’s living a little bit further into the future, which has downsides by the way, because I think that in some ways, I don’t necessarily enjoy the moment quite as much as some other people will do. And so I think there’s a bad side to that as well, that I could probably take some lessons from other people, well not probably, I could definitely take some lessons from other people to just enjoy the moment a little bit more.

So I think that’s one of them. But I think the other thing is this, and I’ve been thinking about this actually with COVID, is one of the things about playing poker that’s super clear is this idea that there are just so many different ways that the future could turn out. So if you think about if someone’s familiar with the game, Texas Hold Em, you got dealt two cards and then three cards come down in the middle, so that’s called the flap and so if we think about a deck of cards, it has 52 cards in it, and now I’ve got the two cards that I can see in my hand, so that’s two that I’ve seen that I know about, and then these three cards that have come down in the middle, so that’s three that I’ve seen and I know about. So that’s a total of five. So five out of the 52 cards I have laid eyes on, I’ve laid… I know what they are, I’m not just looking at the back of them, so now when I’m thinking about the next card, that fourth card that’s gonna get dealt, that comes down called the turn, there’s 47 different cards that could hit that are all possibilities because I’ve only seen five. So all 47 of those cards are possibilities. Now, that doesn’t mean that I’m holding 47 futures in my head at once, because I can chunk them, so I can think about like, well, what if it’s a club? So I get to chunk sort of all the clubs together or I could think about what if it’s a face card? So that would be another thing I could do, or what if fits in a category, which poker players refer to as a blank, and a blank means that it’s probably a card that has no impact on the outcome of the hand. So that’s called the blank, so I sort of chunk them into these things, but I’m probably holding five or six of those chunks in my head, and now whatever I do at this moment, I have to plan for literally any of those to occur and I have to think about how do my actions right now sort of affect what’s gonna happen depending on the types of things that these different chunks of cards that could be coming down. And I think you just get really used to when you’re playing poker, this sort of way of living of sort of holding all these things in your head at once and not feeling like you need to know exactly which one’s gonna hit because honestly, you can’t. You can’t know and you sort of get okay with that, and that balancing act, and I’ve seen in this environment with what’s happening with Coronavirus that I find that that’s something people are struggling with. They want people to tell them for certain what the future is gonna look like. They wanna know what next month is gonna look like when we can hardly know what next week is gonna look like, and I think that that’s just really hard to teach people. You can do it, but I think it’s hard without the experience of really, really seeing that over and over again. And I think it goes into that sort of the probabilistic thinking that I’m really trying to help people to understand. It can be taught, but I think it’s really, really hard to get people to sort of live in that space.

Sean: I feel like someone just to just cut a hole in my skull and just gave you a little preview into my mind as well, so that’s exactly really helpful for me playing out those multiple universes at all times. Any other pitfalls along with that besides not being in the moment that’s someone who experiences that can think through?

Annie: Yeah, I think that with all of the stuff you’re trading off some short-term benefit for some long-term benefit, right.

So if I… Long-term, I think it’s not good to not be thinking about all the different ways that things could turn out good or bad, you know, but short run, I don’t know, maybe you’re happy or if you have rose colored glasses on. Maybe you’re happier if you think next week it’s gonna be great. Maybe you’re a little bit happier if you imagine that everything’s gonna be fine and there’s gonna be a vaccine ready for everybody in October, but I don’t think that that’s good for you in the long run, but I think in the short run, maybe you’re a little bit more anxiety-free, Maybe there’s some more happiness that comes from that, and we should consider that those trade-offs really aren’t real trade-offs and that if I’m saying, Well, I think that the long-term advantage to thinking this way drastically outweighs whatever you might be trading off in terms of short-term contentment or something that I may be imposing my values on other people. I maybe just saying that the things that I value in terms of what I’m trying to get in the long run are more important and I wouldn’t really wanna do that because those are my values and my goals, and I think that I’m willing to trade off whatever sort of anxiety and sort of exploring the dark side that might come with that type of feeling in order to get what I feel are very significant long-term gains in terms of my ability to improve my decision making and sort of what my long-term outcomes might look like, but that’s what I care about. 

So, I’m trying… Obviously, I write the books that try to convince people or convey, I should say, convey why I think that this is a really, really productive way to live your life, and why I think that you should that long-term living out your goals on a… Thinking about your goals on a longer horizon is gonna be really beneficial to you in terms of having great results, but what I imagine is that people who read my books are somewhere in the space of perhaps already thinking that way and trying to figure out how to execute on it. And some people, I may move to this type of thinking through the information that I’m communicating, but some people maybe are just happy, bopping along, and that’s okay.

Sean: No, thank you, Annie. That was a very honest and truthful answer there, so I really do appreciate that, and then one of the things a lot of people are navigating right now is that uncertainty. You’ve really helped me with that. And the way you did that was just kind of helping my ability to understand the difference between complex versus complicated scenarios.

I would love for you just to hit on this for a minute, even just to refresh my thinking here.

Annie: So here’s a difference between complex and complicated. I think it’s a good way to think about what type of decision you happen to be facing. Complicated is like chess, where there’s a very… There’s a much smaller influence of luck and the information is available to you. So basically what you’re doing there is you’re… Technically, from an academic standpoint, what you’re really dealing with is risk that there’s some spread to the outcomes depending on what you do, and you’re trying to figure out if, given whatever decision I make, I can identify perfectly what the possible outcomes are, and I could actually assign probabilities to those. So that now it becomes a question of to what is the downside that I’m willing to bear given any option? How much variance am I okay with for gains and expected value? All this is kind of what you’re thinking, but that can be complicated to calculate. In other words, it could be a really, really hard math problem, you can kind of think about it that way, but the idea would be that it’s solvable. 

So you can think about that like chess is an incredibly complicated game for a human being to actually solve all the way to the last move, but we know that computers can do it. So even though the calculations are really complicated, the actual endpoint is knowable, and what’s interesting with complicated problems is that if I show you an end position in chess, you can kind of calculate what… You could actually calculate what are the different ways that you got there, right, so you could imagine how the game could go from the start and figure out what the possible paths were to get you to the position that you’re looking at.

So that’s complicated. Complex is when we add in a strong influence of luck, but also ignorance, so ignorance technically from an academic standpoint is when you lack knowledge that allows you to really perfectly identify what the possibilities are or what the probability is, obviously, what the probability is, if those possibilities are, because you’re not even quite sure what those are. So once we add in kind of luck and imperfect information into the mix, what happens is that we get into complex decisions, which basically means that your forecasts are subjective, there’s some sort of if you were an omission all knowing being, you would know exactly what the perfect answer was, but you’re not omission and you’re not all knowing, and so you’re working in this imperfect space trying to tease out these problems that really don’t have a solution, not one that is knowable. That’s kind of the difference between those two.

So I really try to help people with complex problems where there is no answer, it’s not like flipping a coin where you know that it’s gonna land heads 50% of the time. When you’re looking for different people who are trying to think about hiring a head of sales could have four completely different opinions about that head of sales. Would an omission being know exactly what the right answer is? Sure, but we’re not omniscient, so we don’t. So those become much more complex problems…

Sean: Oh, Annie, that is so helpful. You really do help anyone who picks up your work understand those complex problems. The first book was Thinking in Bets, your new one, How to Decide: Simple Tools for Making Better Choices.

One final question before we get you linked up with everything, if you were gonna sit down with one person, dead or alive, not a family member or a friend that you could have the microphone, who would it be? Who would you spend an interview, an evening interviewing, who would it be?

Annie: That’s a very interesting question.

I have so many answers, I don’t… It depends on whether I’m right in my topic space or not… You know what, honestly, I know I probably just Daniel Kahneman. I just think that he’s so incredibly smart and he has so much to say, and I have learned so much from him, and just being able to sit and talk to him and explore his mind, but then otherwise… But I can also say like, … would make a really interesting interviewer. I could go back to… Do I get to put Antigone? She would be really cool to interview. Yeah, that’s… Oh god, there’s so many people.

That was such a bad answer, but…

Sean: No, no this shows who you are and pulling from so many different fields and diverse backgrounds of people, so it’s actually probably more clarity than you think, but Annie, I really can’t thank you enough. Like I mentioned a minute ago, the book is How to Decide: Simple Tools for Making Better Choices, anything else you wanna leave the listeners with? And just so you know, I also learn a link up Alliance for Decision Education, ’cause I love what you do there.

Annie: Thank you so much, I really appreciate that. That’s awesome. 

Yeah, no, I hope people enjoy the book. I had a both fun and torturous time writing it, just because all books are hard to write, but I hope people enjoy it. I felt like there was a … of books that gave you real tools to solve some of these problems. There’s lots of amazing work out there on the foibles, like how we go wrong in our decision making, and I just really wanted to write a book to  hopefully show people how you can go right in your decision making, and I hope that’s what people get from it, I hope people enjoy what I put out into the world.

Sean: Well, I certainly do. So, Annie Duke, once again, I cannot thank you enough for joining us on What Got You There.

Annie: Alright, thank you so much, Sean.