Gary Klein

Curiosity

[00:04:07] Sean: Gary, welcome to What Got You There. How are you doing today?

[00:04:10] Gary: Doing very well, thank you. Thanks for inviting me onto this podcast.

[00:04:14] Sean: No, anytime I get to talk with an expert and one of the leaders in their field, especially around things I’m really intrigued about, which is decision-making, I get really excited. But I would love to know, you’ve uncovered so much and you’ve learned so much, but is there a mindset of yours that if you could give to anyone starting out in their career today, you think would be extremely beneficial for them starting out?

[00:04:39] Gary: Oh my gosh, that’s a tough one. What’s the mindset? I’ll tell you what comes to mind. What comes to mind is to be curious and it’s trivial to say, it’s banal to just be curious. However I think coming out of graduate school, a lot of that curiosity is dampened because we’re supposed to be careful. We’re supposed to be rigorous. We’re supposed to collect data in a careful way, in a controlled way so that there’s no mistaking anything. And if the data aren’t coming out the way we expect, it’s very frustrating, but it also is an opportunity to wonder what’s really going on here and to be curious, and to try to dig deeper into the data, into the phenomenon, because you can only collect data carefully about things you already understand very well. 

But when things aren’t going well, when the data are showing some anomalies, that’s an opportunity. It’s not a pleasant opportunity. I’ll tell you as someone who’s collected more than my share of data, that it didn’t turn out the way I expected. It’s always very frustrating. Okay. The study didn’t work and it’s due in for a couple of days, then I’ll look at the data and I’ll say, of course it could never have turned out that way. How could I have been so simple minded? So it’s being open to the data and open to the observations rather than locking into the methodology of the rigor.

[00:06:21] Sean: Was there an experience or maybe even a mentor that helped just cultivate this curiosity inside of you? Because I have to imagine when you were coming out, it was the same thing, extremely hard to adapt to that mindset.

[00:06:34] Gary: I can’t, I didn’t have that kind of a mentor because I came out of a tradition that was very focused on careful collection of data and specifying the methods and being very rigorous in the analysis. And where did it come out? You know, where it came out, it came out accidentally. It was the first study we did with firefighters and we had a hypothesis about how they wouldn’t, … they didn’t have the time to compare all kinds of options. So they must just be comparing a few options. And we had this hypothesis and we had funding from the US army. And then they kept telling us, no, we don’t compare any options.

We had just looked at a situation and you know what to do, and we heard it again and again, and we kept beating against it. We were sure they had to be comparing options and they weren’t. And it was sort of a moment of desperation to say, forget about our hypothesis. Let’s just look at it from their perspective, how could they be doing it? What could be going on? And that changed the whole perspective. Instead of trying to force them into the hypotheses, what we started with was phenomenologically trying to get inside their head and see how this could be happening. To me, that was a fundamental mindset shift or phenomenological shift. That really changed the way the rest of my career went.

[00:08:35] Sean: Were you aware of the impact that had at that time for you? Could you zoom out and understand, you know what, this has fundamentally shifted how I approach things?

[00:08:46] Gary: No, I wasn’t aware at all of the six month project and we had to give the data, analyze the data, work cooperating, and the firefighters hadn’t been cooperating. They hadn’t been telling us what we wanted. And it was like month five going over the protocols. And I said, you know, here’s what I think is going on. And I told my coworkers about it, Roberto Calderwood and Clinton Sirocco and he said, yeah, that looks like what’s happening. So it was an immense feeling of relief that we would be able to write a final report. That’s how narrow my perspective was wanting to get his final report out, that we can finish the project. I had no idea what impact it was going to have.

Habit Routines

[00:09:37] Sean: We’re going to dive a lot into your specific work. I know you mentioned the firefighter instance, there’s going to be a lot to uncover, but I want to get inside your head just a little bit more. And there must be things that you do pretty consistently or have done consistently throughout your career. We can just call these habits routines. Is there anything like that that you’ve stuck with for a really long period of time that you think has helped accelerate your career path?

[00:10:01] Gary: You make it sound much more deliberate and much better thought out than it ever was. Because in fact, there was no career path. I came out of graduate school, not even thinking that I was going to be a researcher. I hadn’t published a single article when I got my PhD and I was in a research program and I worked with two different advisors and I hadn’t published anything. So I didn’t think of myself as a researcher. And so I didn’t say here’s what I want to achieve. Here’s what my career path is. None of that. And then I took my dissertation and I was teaching at St. Lawrence university in upstate New York. 

And I had some time there and I said, well, let me just write up my dissertation. It was on “short term memory”. I was the only one in the department doing that kind of cognitive research because don’t forget this was 1969 before cognitive psychology had really entered into the mainstream of university research. So my advisor was kind enough, Jim Voss was kind enough to agree to be my advisor, but that wasn’t his area. It was no one’s area. So I did this study. I had the data, I had results that came out well, and I said, I’ll write it up. And maybe I’ll get a publication. 

And then I can just go on with my career being a college professor and hope that works. And so I wrote it up and I said, where am I going to send it? And I thought, why don’t I just send it off to one of the best journals in the field? They’ll reject it. But they’ll probably have good criticisms and I can work my way down until I find the right level. So I sent it off to the journal of experimental psychology, and I waited for the rejection and I knew it was going to hurt, but I was preparing myself and to my astonishment, they accepted it with no revisions.

And I thought, well, maybe I can be a researcher. And so that was a formative moment for me in changing my professional identity, that possibly I could be a researcher. Now, I think it’s a mistake. And looking back on it to say, I want to be a researcher. I don’t like that as an identity because it focuses on what kind of person I want to be. I think it’s much more productive to just be curious about things and say, why is that happening? And so you do the research because you want to know rather than because that’s what your role demands, and you’re always looking for what’s the next piece of research I need to do in order to be a researcher.

[00:13:25] Sean: Gary, I think that the beauty in your answer there is that exploratory type nature opened for some serendipity, as opposed to having that staunch career path. So I actually love that. I wish more people had that approach exploring their curiosities and letting things unfold as they will. 

Expertise

Gary, so you mean to tell me we can’t take a handful of college students just put them in the laboratory and can get real world results there?

[00:13:49] Gary: Yeah. I get a lot of arguments from laboratory researchers when I say that and they say the laboratory is the real world too. And it is, but it’s a very small part of the real world because you wouldn’t have those college students, you wouldn’t let them examine you and take care of you if you went into a dentist’s office, would you? 

[00:14:10] Sean: Absolutely not.

[00:14:17] Gary: If your home was on fire, you wouldn’t want one of those college students to be in charge of the crew that shows up to put out the fire and you wouldn’t want to give them anything important. And in fact, one of the constraints of the laboratory paradigm is that in order to get statistically significant results, you want to have a big difference between the experimental group in the control group, but you also want to have reduced variability so that the statistics will come out to show what you’ve found.

So you want to reduce variability. And one way to reduce variability is to make sure that everybody has the same amount of experience on the task. And how do you control that? And there’s one easy and powerful way to make sure everyone has exactly the same experience in performing the task. And that is to give them all a task they’ve never seen before.

So they all start out from zero. So essentially you’re studying novices and you’re doing it deliberately. And because of that, you miss the impact of expertise. And that was what the laboratory researchers were missing. That the people that the natural district decision makers were studying had lots of experience and could use that experience. And, that was a critical factor that wasn’t being captured in the laboratory.

[00:16:02] Sean: That’s one of my big holdups. I look a lot into behavioral economics and they’re these experiments always saying how irrational we are, how poor we are in decision-making and then you think about it. Like there’s a clear difference between a college student who you’ve given 10 hours of training, opposed to someone who’s been doing this for 25 years. I mean, they’ve got this fingertip feel that’s hard to calculate. And so I appreciate your work bringing some of this light. I would love to know, through your lens, the difference between a novice and an expert though.

[00:16:33] Gary: Right. You think that I would have a good answer for that, but I don’t. Because there’s no gold standard for who is an expert. There’s been a number of papers published on the nature of expertise and what experts can do, what they can see that’s different from a novice. Experts notice things that novices don’t notice, experts can anticipate what’s likely to happen next and novices can’t, experts rapidly size up what’s going on in a situation. So they know what to do in that situation. They know if their expectancy is violated. They can quickly see that this isn’t happening the way it’s supposed to, something must be wrong. And maybe I have the wrong interpretation. 

So there’s a variety of ways that experts differ from novices. What’s hard is to know who is an expert and I identified 10 different criteria. None were foolproof. One way to identify an expert is someone who’s had lots of years of experience. There are too many people who have lots of experience who never really developed expertise because they’re not reflecting on their experience or learning from their failures or getting better.

So years of experience doesn’t do it. It could be that someone is an expert because they have a track record of success. But it may be that they were lucky. Just because someone has had success in the past, maybe they were just fortunate and maybe it was different circumstances than the one you have now. It could be that it’s someone who carries himself with confidence and inspires confidence. There are too many people who have learned to inspire confidence who really don’t have expertise. So together, these kinds of criteria make sense, but each one of them has its own limitations.

[00:18:59] Sean: Yeah, I think that’s so important to understanding the varying degrees of all of these things. One of the things you mentioned around expertise is just really analyzing their own knowledge as they progress. So you have a 30 year career and it’s not just one year repeated again and again, you’re advancing during those careers. What did you actually do to let’s just call it like distill down some of the lessons you were learning so that you were advancing on a steep trajectory.

[00:19:25] Gary: I didn’t do anything. So the experts, the study, they have an area that they want to get better at. If I’m a firefighter, I want to become a better firefighter. And one of the criteria I use folks for who’s an expert is I’ll ask them, tell me about the last mistake you’ve made. And if somebody says, I can’t think of any mistakes right away, I downgrade them because experts readily think of their mistakes because their mistakes haunt them, their mistakes eat at them. And they’re trying to figure out what I should have done? 

Because they know that there’s a skill they want to get better at. The reason I’m disqualifying myself is there wasn’t any skill I wanted to get better at? I wanted to study experts and I guess maybe that’s the skill of trying to,… okay. So maybe that’s it. I wanted to get better at understanding experts. And when I would do interviews with experts after the interview, I would think I should’ve asked that question. Or I’ll go over the transcript and say, I can’t believe that I didn’t put these pieces together and follow up in that way. So, or what would have happened if I had tried this method instead of that method. 

So I guess maybe that’s the way that I’m trying to get better. And I still try to get better, to improve my skills at capturing expertise. There was one incident I had, it still bothers me. It was really a stupid incident. We had a sponsor who made soup and they wanted us to study their consumers to understand their consumer decision-making. And so we had a few of us show up at a test lab. And there were people who were recruited, standard recruitment and they came in and there were a bunch of cans of soup.

And we said, pick one of these cans, you’re going to have lunch here, pick one of these for your lunch. And they would say, that’s the one I liked. So we would take that can off, we would make soup for them. And they would eat the soup and we would interview them as they were eating the soup, so this is really very, very trivial. And I remember this one woman, it was her favorite soup and she really was relishing it. And I said, how could the manufacturer, how could it get better? What could make it any better? And she said, I can’t think of any way. It’s perfect the way it is. And I said, it’s always perfect.

She said, every spoonful is perfect. The interview finished, I didn’t get any suggestions from her, but as we were driving back with the others, I realized that’s what she said. But as I watched her, she would take a spoonful of soup. She’d look at what was in the spoon. And then sometimes she would shake it out or take it at another spoonful. And I should have said, why did you shake out the last spoonful, what was missing? That was the wedge. It was right in front of my eyes, but I was so focused on asking my question thatI wasn’t watching what she was doing. It still bothers me that I missed that. And it was just a lousy bowl of soup that she was eating.

[00:23:20] Sean: Any idea why that still bothers you?

[00:23:17] Gary: It bothers me because I should’ve been better. I should have been watching. And it’s a constant reminder not to be so locked into my interview guide that I’m not watching and listening to the person that I’m studying.

Great Decision Makers

[00:23:38] Sean: Gary, I know you’re just filled with stories like this. I’m wondering for you, do you have any great stories around someone you would deem as an exquisite decision maker and how they actually executed great decision making in the real world?

[00:23:52] Gary: A great decision maker? Well, one of the people that just awes me is a guy named Robert Schmidle and he’s a former Marine. He retired as a three-star Marine and was a pilot. And he had reddish hair that kind of stood up. So pilots have to have call signs. So his call sign was “rooster”. So that’s how everybody knew him as rooster Schmidle. And I met him when he was a Lieutenant Colonel and there was a Marine program in California and they were trying out some new electronic equipment that was going to allow a different organizational structure.

And this was a program. There was a big test coming up in about two or three months. I got called in to be part of that test. And frankly, the preparation wasn’t going well, and the person in charge of making that happen, just wasn’t succeeding and that person had to be removed and sent to another assignment and Schmidle gets brought in. And now you have this malfunctioning group of people. It was a large cast of characters. I would say 30 or 40 people in a command post, a futuristic command post with technology nobody understood that was going to have implications nobody could anticipate. And there was going to be lots of data collected and lots of visibility and Schmidle is in charge of making it happen.

And I watched what he did and he came in and he didn’t give any speeches. And he didn’t talk about game plans or any kinds of plans. He just wanted to understand how this command post was working. So he said, do your job. There were some test problems they were working on and they went over to one person and he said, okay, so you and they all had a computer screen in front of him. And he said, so tell me what your job is. And this person said, here’s what I’m doing. Here’s what I’m supposed to achieve.Schmidle said, okay, who do you get information from? And the guy said, I don’t know, somebody back there.

And Schmidle said, and you get the information and you do something with it, you analyze it and you send it off. And the guy said, right, who do you send it to? The guy says, I don’t know, I just press send, and that’s the end of it. Schmidle said, okay. I’d like you to get out of your chair and let’s go around and find out who is sending you information. And let’s understand who they are, what they’re doing, why they think you need it. And let’s go around and see who’s the people who were sending you information. And so we made our rounds to try to find out who was sending this guy information. Everybody’s watching us and listening to what’s happening.

And then we had that sorted out. Schmidle said, now you send the information, who do you send it to? And the guy said, you know, I’ve got some names on our roster. Let’s talk to those people. Let’s find out who they are and why they need it and how they’re using it. And so we started and we did that for a couple of days. And in a few days, everybody understood how that unit was supposed to work. And that’s different from say, here’s a wiring diagram, or here’s a mission statement, or here’s a flow chart. It was personalizing it so people could really get a sense of how this whole unit was supposed to make decisions together. And Schmidle didn’t know the answer. He wasn’t trying to lead anybody. He was trying to discover it with them and put them into a curious mindset. So that was one of the most dramatic things that I could remember seeing.

[00:28:29] Sean: Gary, I love that story. I’m wondering for people like Schmidle before you were talking, just being able to analyze your own decision-making so you can advance, how have you uncovered that the best decision makers take this knowledge, distill it down and are able to pass it down to those earlier in their organizations, their teams, just so those people can learn that type of knowledge earlier in their career.

[00:28:55] Gary: Right? How do you disseminate it? How do you package it? I don’t think they have developed a blueprint because I think blueprints are for things that are literally to be nailed down. And that’s the opposite of these voyages of discovery. So I think what’s going on here is you have people who are encouraging and engaging in a discovery process for the people they’re helping and for themselves so that others can learn what they need to learn.

And still you can look at them and see, are they understanding, are they comprehending what’s happening or are they confused? Because, there’s a way that people just signal us with their expressions. No, they just have a quizzical look that says I don’t quite get it. So you need to be sensitive to people’s reaction to know, do I have to change my instruction?

Do I need to add more information? We don’t want to over explain, but I don’t want to under explain. So it’s really forming those kinds of relationships that allow you to enter them into a real dialogue with people. And then you tailor what you tell them or what you let them ask. When you say something’s bothering you, what’s bothering you and you give them an opportunity to ask you what’s bothering them.

[00:30:41] Sean: These last two stories, it sounds like relationships play an important role in this development. Am I correct in assessing that?

[00:30:52] Gary: Right. And the ability to take other people’s perspective.

[00:30:55] Sean: You mean, we’ve got to listen to other people now, Gary?

[00:30:57] Gary: I am just doing a project now about making diagnoses in emergency departments. This is a project that is being led by Johns Hopkins university. I should guess Johns Hopkins is a leader of this effort and there’s this one incident where somebody comes into an emergency department and she doesn’t know why she’s there and she doesn’t think she needs to be there, but her family said, no, mom, you need to go to the emergency department. What was happening was she was having strokes, but she didn’t know it. 

And so she didn’t want to be in the emergency department. And she said, I’m perfectly fine. And she denied any kind of disability and the physician who was examining her glanced over to the two family members and they rolled their eyes. And he knew that regardless of what she was saying, there was a real problem there. And so he said, okay, thank you. Ma’am just wait a second. And he took the family members out into the hall where he could talk to them because they didn’t want to contradict her while she was present, because it was making her very agitated and very angry. 

So they weren’t going to contradict her in front of her. And this physician picked up that something was happening just because he saw them roll their eyes. But how many times have you gone to be examined and the physician or the physician’s assistant is just staring at a computer screen? And isn’t looking at you at all and isn’t picking up those kinds of non-verbal cues to be followed up on.

Pre-mortem

[00:32:56] Sean: Yeah, it’s so true. I try to avoid doctors at all costs, but at any time I’ve gotten great feedback. It’s like they’re truly there, in the moment. They were tuned to what I’m saying, their understanding of the patterns of my voice to uncover those little things. And, this just makes me think about one of the things you brought to light and that’s around the pre-mortem.

I know a lot of people have heard of the post-mortem where you analyze things after the fact, but you were talking about talking with the family member and they were afraid to bring things up in front of her. And it makes me think of the pre-mortem where you’re doing some of these things prior. I would love for you just to dive into the pre-mortem. Because this is something I’ve implemented has been incredibly helpful for me.

[00:33:40] Gary: Right? Pre-mortem is a source of surprise and confusion for me because it’s got legs. People like you have heard of it and are using it and people all over are using the pre-mortem. And most of the time they don’t even tell me. And usually they don’t tell me and it’s just sort of taken off and I never expected any of that. The pre-mortem is a simple exercise. People know what a post-mortem is. If a patient dies, you do an autopsy to find out why the patient died. And usually you can discover the cause of death or do a reasonably good job of it. 

And so the medical staff gets smarter because they get feedback about what went wrong with this patient. And, it’s also useful for the family because the family finally learns what it was that led to the death of their loved one. And if it’s something interesting and unusual, they could write it up as a note and it could be published for the community. Everybody benefits from a post-mortem, except the patient, because the patient is dead. So it doesn’t help the patient.  And I applied at that to research projects we would do. 

And if the project went poorly, we would gather together in a conference room to say, what went wrong? And, you know, maybe what could we have done to prevent it? And maybe we learned something, but it was too late for that project. That project had already failed. So I said, why don’t we do that session at the beginning rather than the end. And when we have the kickoff meeting for a new project, we say, here’s the project.  We’re all excited. Now let’s take 20 minutes. And imagine that the project has failed. 

Not just imagine that it’s failed, I have a crystal ball. And actually I do have a crystal ball someplace, just not with me. And then the next thing that I’m looking into a crystal ball, and it’s now six months from now or a year from now. And the image in the crystal ball is as clear as you want. And it shows this project that we’re all so excited about, this project has failed. There’s no doubt the project has failed. And the people on the project are frustrated and angry and embarrassed and irritated with each other, but there’s no going back. 

The project has failed. We know that everybody in this room is getting ready to start, this project has failed. That’s what the crystal ball is showing. Now everybody takes two minutes and writes down all the reasons why the project failed and everybody starts writing like crazy. And then two minutes are up and I only give them two minutes. And I say, now let’s go around the room and write down all the reasons people came up with. And it’s amazing. The kinds of things that we learn when we do this pre-mortem. And it works because the mistakes people sometimes make, when they do a pre-mortem they’ll say now what could go wrong? 

And there’s so much social pressure not to come up with anything or people aren’t thinking of anything that could go wrong. They’re not even imagining it. So you have to change that mindset that it has failed. And with a typical review at the end of a kickoff meeting, there’s a lot of pressure not to be critical because people will be angry with you. So there’s all this pressure to say, no, it’s a great plan. We did a great job. Let’s get started. Whereas the pressure in a pre-mortem it’s a competitive pressure is to come up with important problems and barriers that others haven’t identified. And that’s how you show how smart you are. 

So there’s this competition going, but, to identify types of things that can go wrong and we’re continually amazed by the types of discoveries, the types of thoughts and speculations that people come up with. And we’ve done research and we find that the pre-mortem seems to be a much more effective technique at reducing overconfidence. That’s one thing it does. It also prepares people for possible problems. Then when we go back and we say, what can we do to make a better plan using the pre-mortem? And the pre-mortem also creates a culture of candor to some extent. This is where a team that can criticize ideas and possibilities. And we’re not going to be afraid to say anything that might upset people.

[00:38:56] Sean: Yeah, I think too many people are familiar with that environment, with those very draconian leaders who the second they say any type of contradictory evidence or anything that person’s going to shoot it down, which is why the pre-mortem is so good. And another thing I’ve seen this help in different organizations is people earlier in their career, younger than organizations who usually don’t voice things, all of a sudden they bring these insights that no one would have ever thought of. 

And it would never come to fruition unless there was this platform that was able to do that. So yeah, you’ve got some great tools and resources online around the pre-mortem. I’ll make sure that those are linked up, but one of the things we were just talking about is great decision makers. They’re able to accept that contradictory evidence and they’re able to examine it. I’m wondering what you’ve uncovered about this. Around good decision-makers taking information that might hurt their ego a bit to get the better decisions.

[00:38:48] Gary: Right. Good decision-makers still have an ego and there’s things with themselves. So it’s not that they’re the most humble people around. There is a certain amount of arrogance.

[00:40:06] Sean: Is that actually an essential component of great decision makers? Like they’ve put in the work, they’ve done the work and they are able to learn where they’ve got extreme confidence and there’s a reason for it, or is that just a bias they probably have?

[00:40:24] Gary: Probably part of the constellation that they’re not afraid to try something. They have enough confidence/arrogance to say, all right, let’s see how this works. Let’s give it a shot. And this may do the trick, or we may have to adapt it, and we’re good enough to make those adaptations. So that kind of confidence is probably necessary to go through the discovery, to go through the exploration that others who don’t have this self-confidence just sort of hang back and are afraid to try something newer or to suggest something one might be bold or might be unpopular.

So I think that’s part of the dynamic. But I think the great decision makers want to have successes. And so when they’re doing the work, it’s really about the work coming out well, rather than them looking good. And that’s why they’ll tolerate and welcome a contrary view because it gives them things to think about. And because they are confident, they don’t feel challenged when people come up with criticisms. 

They’re not crushed by it. And so I think it’s that kind of mission focus that, let’s just be candid. Let’s just try to understand what’s working here and what’s not working, and then see if there are ways we have to adjust or maybe we have to change what we’re trying to do. Maybe we have to make a radical revision. They’re open to that kind of improvisation.

[00:42:20] Sean: Does this kind of fall under the category of some of the things you’ve done a lot of work on and that’s recognition primed decision-making essentially how experienced people make decisions. Is this what you’d classify under that category?

[00:42:35] Gary: Not entirely. I think recognition primed decision-making is at the core of a  lot of this type of performance is that people look at situations, know what’s going on, know what to do about it. And so they don’t have to spend a lot of time and energy sorting out what’s going on. They already see it. But they’re also mission-oriented. So if they get some contrary data, they’ll say, maybe I’ve misinterpreted. Let’s go back and let’s rethink it. So they’re not fixating. They’re not locking into their initial impression, but because they have such good intuitions and such experience, they can proceed and then they can also be open to anomalies and to answer problems and see when they have to adjust.

[00:43:34] Sean: Believe me, we’re going to circle back to intuition. I know you’ve done a lot of work there. I’m really intrigued by that. But you bring up that just the experience makes me think of some of the studies around chess, grandmasters and they can look at a board and they can just understand so much where they essentially look at less, but see more. And I’m wondering, how do you analyze this in the real world? Does this just come down to filtering where you understand the 99% of stuff that’s irrelevant in these situations? And you can just highlight and focus on the key metrics. Is that what it’s about?

[00:44:08] Gary: That’s an important attribute of it and an important aspect of it. And that’s probably something that people could measure is what kinds of information do you want? And that’s even a question that you can ask people. And my friend, John Schmidle, a former Marine, uses this when he runs decision-making sessions. He’ll ask the students, what do you want to know? At this point, what do you want to know? And based on the question they ask, he can get a pretty good idea of whether they’re really understanding the dynamics of the situation. 

For example, it might be a tactical decision and somebody might say yeah, I’d like to know what’s the weather forecast. And it turns out the weather forecast really doesn’t matter. You’ve got this enemy unit in a place that you didn’t expect. And now you’re trying to figure out what to do with it. And what you don’t know is what they are planning to do next? That’s the critical thing that you need to know, not if it’s going to start drizzling the next day.

Asking Great Questions

[00:45:20] Sean: It makes me think actually one of the best ways to get to great answers is great questions. Are there any other really good questions you see exquisite decision-makers tend to ask of themselves?

[00:45:30] Gary: None come to mind. I even have a negative reaction to that question.

[00:45:35] Sean: Dive, dive further on this then 

[00:43:54] Gary: Which is, I think if we try to come up with a standard set of questions that are good questions, that’s the opposite of being curious about what’s going on in a situation. So even if I could come up with a checklist and now I’m going to start thinking about what would be on the checklist, I don’t think it would help people to have that checklist I think would get in the way.

I don’t want people to say which of these should I be asking? I think they want to be in this situation, looking at the incident and trying to understand what’s happening, what they don’t understand, what they need to know more about. So when I do cognitive interviews, I never want to,..  Sometimes I have to go into an interview cold, but I like to have an interview guide. And these are the sorts of the areas that I want to explore. 

And we’re just getting ready to start a big cognitive interviewing project tomorrow. So we worked on an interview guide, but it’s just a guide. It’s about the topics that we want. And I know that once the interview starts around tough cases and critical incidents the guide is going to be over here, but I’m going to be asking about the incident and how it unfolded and why it took certain directions that maybe weren’t expected, why it became challenging, why it became confusing, what was happening that was unexpected. And just go from there and forget about the interview guide.

[00:47:18] Sean: I think just this far, a 30,000 foot view of this conversation, explore your curiosities, don’t expect the blueprint. You use that as a guide book, be open to exploration there and really analyze the decision you’re making. It seems like these are some of the key themes which is good because I feel like too often based on my questions, you can see, it’s like, we’re looking for, Hey, what’s the five-step plan. 

Skilling Up Expertise Level

And I think most people have established that life doesn’t work that way. But I’m wondering, you’ve gotten to see so many of these wicked environments and where people have been on these really steep learning curves. Are there any commonalities of people who have really just skilled up their expertise level? An extremely quick pace that you just bring to light?

[00:48:05] Gary: Okay. But before I answer that, let me go back to the previous discussion about wanting the five steps. Yeah. People do want the five steps they want to get that kind of a blueprint. I want that kind of blueprint. We all feel more comfortable with it. And that’s the tendency to proceduralized how we work. That’s opposite from being curious and then, and being open and flexible is. If I’m open and flexible, what if it fails? What if it falls flat, then I’m stuck. So, there’s a risk there.

There’s a nervousness and anxiety about not having five steps or a clear blueprint or a clear set of steps to follow to carry out a project. That’s anxiety producing and that gets in the way of curiosity. So I share that desire for procedures and blueprints, but I know that it has these negative consequences. Now you’re asking about wicked problems, tough cases, and people who can will themselves to get up to speed more quickly. And how is it done? Is that that’s what you asked? 

[00:49:30] Sean: Good memory there. Great way to compartmentalize.

[00:49:33] Gary: So what have I seen? I gave you the example of “rooster Schmidle” getting the whole experimental command post up to speed by the way he conducted it. It wasn’t just an investigation. He had people walking around from one site to the other and meeting people and it was like watching a force of nature. What else can people do to get up to speed more quickly? They can talk to people who are more experienced. They can ask them, they can interview them. They can ask them about tough cases and people often like to tell stories, but yet you can’t just let them tell their stories. You want to probe them. Well, wait a second. You did that.

I don’t see how you knew that. And so you just probe them to get at their tacit knowledge, because when people tell stories, they gloss over the tacit knowledge that they have. And the story gives you an opportunity to say, timeout, why did you think that, that was your hypothesis? What was driving you towards that hypothesis? You don’t just have to listen to this story. You can be an active part of them unpacking the story. And sometimes maybe even helping them realize how they made decisions that they didn’t understand at the time. So that’s one thing you can do. 

Another thing is to use scenarios. In some situations, there are good challenging scenarios, and you can put yourself through scenario training and we have a technique called shadow box, for example. That is a scenario, largely a scenario based approach where people are put into tough cases and are trying to handle difficult decision points about what to do, what information to collect, what goals to pursue. Then they compare what they would do, to what a small group of experts who have gone through the same scenario, what the experts would do. But the trainee writes down, here’s why I ranked it. I did it that way. Then you get to see why the experts rank it the way they did, and you understand how the experts were thinking.

So that’s something else that could be done. There’s a variety of things. That people can do. You can shadow somebody, you can just be like an apprentice and go around with them. Which I think is useful for watching them. But you also want to have some time to reflect with them afterward. Why did you put that there instead of there or why you used this map instead of that map or just ask them about some of the details about how they did their job, rather than just trying to be quiet and meek and cooperative.

Nature of Insight

[00:52:44] Sean: Yeah. Gary, we learn from stories. You have a great story and my apologies, I forget which book it’s from, but it deals with cops, a brand new BMW and “ashing” a cigarette. I would love for you to tell the story because I think this really makes it clear for anyone listening to this.

[00:53:00] Gary: This was part of a project we did to try to understand the nature of insight. And if you don’t mind, let me back up and say why I was trying to understand the nature of insight. It was because I had a chart that I would use when I would give talks and the chart had two arrows, one going down, one going up and it was about improving performance. And I said, if you want to improve performance, the arrow going down, that’s what you want to reduce.

You want to reduce mistakes, cut down on errors. The arrow going up that’s what you want to increase, you want to increase insights and expertise. Most organizations primarily focus on the down arrow. It’s all about reducing mistakes, but that can’t be the totality of your work because you don’t want to come home at night and say, I had a great day, I didn’t make any mistakes. I mean, that’s not why you’re being paid. You want to accomplish things. 

But most organizations just think about reducing mistakes and they have ways of doing that. They don’t think about how we can promote insights. So I present this to groups and people would agree, they’d say, yeah, that’s my organization. It’s all about the down arrow. Then they would ask me, what can you tell us about the up arrow about increasing insights? And I said, I don’t know. I’ve never thought about the up arrow. So this happened a number of times and it was irritating enough. 

And it happened once in Singapore, so there was a long, 17 hour flight back from Singapore to the United States. I had a lot of time to stew about it and I said, I should do a project to see if I can come up with any ideas about increasing insights. And so I collected 120 examples of insights, some of them from books, some of them from articles, some of them from interviews that I had done. This example with the two cops came from an interview that I had done for another project.

And what happened was, and the guy who told us about it was the senior cop, senior police officer, and they were driving in a patrol car. They’re stuck in traffic, it’s a red light and just sitting there and waiting for the light to turn. And his partner who was still on probation or a new police officer and out of the academy, not all that long looked at the car in front of him. And he sees it’s a brand new BMW. He watches as the driver takes a drag on a cigarette and flicks the ashes. 

And he says, what, what. Who flicks the ashes in your brand new BMW? Would you do that if it was your car or if you borrowed it from a friend? Would you do that? That doesn’t make sense. Let’s see what’s going on. So they light up, activate their lights, pull the car over. Sure enough, it’s a stolen car. Okay. So the young police officer didn’t know it was a stolen car. There was nothing announcing that it was a stolen car. All he saw was the action of flicking the ashes. And he said, this doesn’t make sense. This doesn’t add up, something else is happening here.

And that became one of the pathways we identified for how insights are. When people arrive at insights, they see contradictions, they see anomalies and that makes them investigate harder. And to me, you’re exactly right. It’s one of my favorite stories. This is like a poster child for the contradiction path. You just saw something that should not have happened and you need to know why.

Shadow Box Training

[00:57:17] Sean: No, Gary, thank you for sharing that story. Yeah. That’s why I love it, it makes it so clear, so vivid in the mind. You were mentioning shadow box training, which you developed and I would love to understand even a little bit more about what you guys actually do with shadow box training. I know this is a company of yours. I think it’d be really helpful for people to understand exactly what you guys are doing and how you go about implementing it. So people can essentially learn through the eyes of experts.

[00:57:43] Gary: So the idea of shadow boxes to help people see through the eyes of experts. And you’re essentially shadowing these experts, but the experts are expensive. Their parts are scheduled. So you don’t want to have to rely on experts. You want to be able to do the training, get people up to speed, get them smarter, give them richer mental models so they can learn from the experts without having to have the experts there. And so this is a technique.

It originally started with some work by a friend of mine, a firefighter, Neil Heights. So the dominant way that we use it is with a scenario. I’d give you a tough scenario and I’d walk you through the scenario, and I’d say, let’s stop here. Here’s four courses of action, rank order which you would prefer to do from top to bottom and write down your reasons. And then we continue on with the scenario. We stop it again. It’s another decision point here. At this point, you have three possible goals. We’re going to identify the rank order of the goal and write down your reasons. 

And then we continue the scenario. We might stop it again. Here’s five pieces of information, rank order which is the most important to the least important and write down your reasons. We’ve also had a small group of experts, subject matter experts, usually about three, sometimes more maybe up to five or seven. They’ve gone through the same scenario. They’ve done their ranking. They wrote down their reasons. So when you say here’s the three goals, here’s how I rank order them then you get to see how the experts rank ordered them. And you want your rank ordering to match the experts because that’s sort of the game-like quality you want, you want that feeling. 

But the important learning comes in when you look at what the experts wrote down as their rationale, what was their reason, and you see, they would notice all kinds of implications, drawing all kinds of inferences that you were never drawing. And now both of you, you’re seeing the world through the eyes of experts because it’s the same scenario, but they’re seeing things that you did. And so that’s the usual paradigm that we have. There’s another paradigm that uses video clips, where we’ll show you a video clip of something going on.

And then we’ll say, now that you see how it ended, let’s go back and you can pause it and put your cursor on anything that seems significant and write down why, and you do that. And then when you’re finished, we go back and show you when the experts clicked, where they put their cursor and why. And now you really are seeing the world. You’re seeing both seeing the same video, but you’re seeing it through their eyes.

Cognitive Transformation Theory

[01:00:43] Sean: Yeah, Gary, what I appreciate about this technique is it makes me think of when you were growing up were, you know, working on those math problems in the math book, and you go back, you can understand how something is done, but you get to truly understand the why, the reasoning behind why the experts make that decision. I think that is a fundamental shift and helps you really understand the different scenarios. One of the things that this just makes me think about is that you have a great paper on cognitive transformation theory. And I would love for you to uncover this a bit more exactly what cognitive transformation theory is, and then dive into some of the things that this helps people shift their mental models on.

[01:01:22] Gary: Nope. I’m not going to do that right now because you have an unfortunate knack of asking exciting questions that make me think of other things that I want to say, and then go on to another question.

[01:01:38] Sean: My apologies

[01:01:40] Gary: We’ll get to cognitive transformation theory in a second, but let me go back to what you just mentioned about working math problems in a math textbook. One of the flaws of math textbooks is they’ll give you a whole bunch of problems that are all based on the techniques that you just learned in that chapter. Whereas in fact, they should have problems at the end that draw on all the things that you’ve learned. So you’ve got to figure out which techniques apply rather than just apply it in a roped way, what you’ve just learned.

And it would be nice if you also had this meta knowledge from experts to explain why you use this technique rather than the other. So you can do a lot more with math textbooks than just this kind of rope mindless drill. So that’s one reaction I have. The second reaction is…  forget about math. What about cooking? I just made fudge for my youngest. He asked me, it was his birthday and he wanted me to make fudge for him which I didn’t want to do because I’ve never successfully made fudge, but my daughter said she’ll help me out, but I’ve always failed. 

So Saturday we were making fudge. Here’s the recipe and you have the recipe, here’s the ingredients, here’s the steps, but there’s nothing cognitive about it to say, here’s why we’re doing it this way, here’s the kind of common mistakes that people make, here’s what you can anticipate, here’s why this step comes before that step, here’s how to tell when this is ready to move on to the next stage. There’s none of that in a recipe, it’s just a list of procedures and rules rather than something much more dynamic. Whereas in fact, cooking depends on RPD and recognition skills to know I’ve stirred this enough and this is ready to move on to the next stage. So this applies in a variety of ways. 

Now, to your question about cognitive transformation theory. This is work that I did with Holly Baxter and the idea of cognitive transformation theory is that if you are going to get better, you’re not just adding more rules and procedures to the knowledge that you have, like items in a warehouse. You’re going to change your beliefs. You’re going to change your mental model. You’re going to discard some things that you believe, or you’re going to modify them and say, okay, this holds only under these circumstances, not under those circumstances.

And that’s the cognitive transformation that we should be trying to achieve. And it’s not simply what you’re learning, but also when you’re unlearning. That there are certain things that you believe that may be wrong or may be only true in a narrow range. And you need to discover what that range is so that you’re not misapplying it. And so the goal for cognitive transformation theory is to encourage people to build richer mental models that involve unlearning and relearning and really changing beliefs, drawing insights, building on insights. 

That seems to be essential even in classrooms that classroom teachers can move from an orientation, here’s the material I’ve got to cover, and I’ve got an hour to cover it, to try to foment insights in the students and have them make their own discoveries. Mickey cheek has talked about this, about this kind of self explanation that we learn best when we make our discovery, rather than when somebody tells us things. So how can you arrange the conditions for the trainees to make their own discovery. And sometimes it means not telling them everything, leaving gaps. So that they have to actively engage in bridging those gaps. And in doing that, they’re building more powerful mental models.

[01:06:35] Sean: Yeah. You mentioned unlearning there. It makes me think of that Sherlock Holmes quote, I’ll paraphrase here essentially. Like your brain should be like an empty attic. What you fill with the furniture is going to be so important, but sometimes you have to discard that old furniture, meaning your old mental models that aren’t going to be helpful. And there’s even like the great I think it’s the storehouse model you’ve talked about. It would be like having a filing cabinet and you could pile on new files, new facts, everything like that. But if every single new fact is going to make that filing cabinet, just messier, meaning your mental models aren’t up to date, new facts and new knowledge is not going to help you. You’ve got to revisit those old mental models. That’s one I just love. It’s always in the back of my mind.

[01:07:15] Gary: Right. It’s not cramming more over into a file cabinet and saying, now I’ve got it because you’re making it harder to find what you need. I have an image that’s always stayed with me. This is from the 1960s. So this is from a million years ago when I was a graduate student at the university of Pittsburgh. And this was in the computer center and people would have punch cards. People don’t know what a punch card is, but it’s like a strip of cardboard and you would punch out the digits from zero to nine and that’s the way the computer was being programmed. And people carry around boxes of these punch cards. That was their program. It was so primitive. I was primitive and that’s the way it was. 

And at one point I was in the computer center and they were cleaning the punch card machine. So they opened it up and they were sweeping all the chads, chads of the little rectangles that got punched out. It was sweeping the chads onto the floor, so they could clean the machine and be ready for the next day. And then a guy had his broom and he shoveled it together so he could remove it. And he couldn’t stop himself. And he reached in and he just lifted up a whole handful of chads. And he said, I have more knowledge in my hands than I do in my head. What you’re having in your hand, are a little pair of cardboard squares, the digit zero to nine. We have nothing here. He thought he had a lot because there were so many of them that it must be important and its significance. 

So yeah, cramming more information in doesn’t always help. And in fact, that’s another problem that people run into when they’re not sure what to do. They say, let me gather more information rather than let me try to figure out what’s going on here with the information I have. So this is a tendency that gets in people’s way to think that more information is going to help. Because the marginal value of each additional piece of information is smaller and smaller, but you have other complications as well. So the search for more information, the desire to cram more pieces of paper into your file cabinet, so you can be smarter, is as you point out, very misleading and unfortunate. 

Tying Emotion In Learnings

[01:09:57] Sean: Yeah, absolutely. Another thing you mentioned a minute ago is when we experience these certain learnings, they stick with us. I always think about this like when emotion is tied to something for me, for some reason, it sticks with me so much longer. I’m wondering within shadow box training, have you guys implemented anything like that, where you try to tie emotion to help with memory and these learnings?

[01:10:19] Gary: We haven’t done it as much as we should. That’s a great point because emotion is, …I believe one of the functions of emotion is to create urgency and create saliency and allow us to properly value experiences that we’ve had. With shadow box training we see emotion, we see stress when people are stuck and they don’t know what they’re supposed to do, or just how to proceed or what’s going on in a situation.

And we see delight when people figure things out. Or when sometimes when we say here’s the way you ranked the options, here’s the way the experts ranked them. And they say, oh, of course, of course. And so that’s actually not a moment of delight, but it’s a moment of insight. Sharing with them, what the experts were picking up on and that stays with you. Those kinds of breakthroughs are very exciting and very meaningful. Yeah, I agree. 

Gary’s Thinking On Intuition

[01:11:33] Sean: Another thing you brought up earlier, I want to dive further into and that’s just understanding intuition and being able to listen to it more. I would love to know just like your latest thinking on intuition.

[01:11:45] Gary: This is a longstanding argument that I have with a leading researcher, Danny Kahneman. He and his community are very uncomfortable with the idea of intuition. He talks about system one thinking, which is very associational, very emotional and system two thinking, which is like Mr. Spock, very analytical, very careful and they find all kinds of heuristics that people use. Because system two thinking takes effort and people usually try to rely on system one instead. And so they found heuristics and they showed that people use these heuristics. Because they put them in situations where the heuristics don’t work and they still use the heuristics. So they say this is about biases since that’s how they’re able to demonstrate the biases. 

And so people will ask me, when can you trust your intuition? And my answer is never, you can never trust your intuition because it might be wrong. That’s why the model that I have of decision-making has two parts. The first part is the intuitive parts where you use associations. You use patterns that you’ve learned to size up what’s happening, what to attend to what you can ignore. You talked about that before, what to expect. And if the expectation is violated, that’s an interesting and important piece of information. What goals to pursue, what courses of action are likely to work. That’s what your intuition is giving you. And so people with experience can do that very rapidly.

So that’s the first part of the RPD model, but the second part comes from the issue, how do you evaluate your intuition? If people, firefighters and others told us they don’t compare options, they use their intuition to tell them what’s likely to work well. How do you evaluate it except by comparing it to another option? And what we discovered is they evaluated it by imagining it. By saying, this is the course of action, let me think through what would happen if I carry it out. How might it work? How might it fail and say they do what we call a mental simulation. They run it through in their minds to see if it’ll work in this context, in this situation.

And if it will work, then they’re ready to make the decision in just a few seconds. If it almost works, then they modify the course of action to overcome the problem. If they can’t figure out a way to fix it, then they say, forget that course of action. What else can I do? So intuition is just part of the story of decision-making, but it’s a critical part. And it’s a part that the laboratory researchers missed because they weren’t studying people who had experience, they were giving the college students novel tasks. And so they had no experience base to draw on. 

Mental Simulation

[01:14:58] Sean: I never understood why iron’s going to be like, so binary, right? Like throwing intuition out the window. No, no. It’s like these things can help encapsulate a total model and help us get to better decisions. So yeah, I appreciate you shedding light on that. Another thing I just found really interesting, you just mentioned is the mental simulation. I used to play a lot of athletics, so I used to do a lot of mental imagery prior to an event and I still do it now. So it almost seems like that is applicable also in athletics, where you’re running through potential scenarios, both the positive, but then also the downside of trying a certain move and things like that. It’s just interesting how these patterns emerge all throughout different domains.

[01:15:39] Gary: Yeah. But I think there are people like you who really want to get better, so they’ll go through the effort. It’s not that you’re going through the effort, you can’t stop yourself from doing this kind of imagining to see, if I did that, how would that look? That’s just part of your makeup, but there are other people who just want to, they figure, I put my time in during the practice session, I don’t have to think about it anymore. And they stopped thinking about it and they didn’t improve. So you asked before about what it takes for people to get better? That’s the kind of mindset that it takes for people to get there. 

Influential Books In Gary’s Life

[01:16:23] Sean: Yeah, it’s a light switch. I don’t know if it’s good or bad. I just can’t seem to turn it off. It’s that kind of just being drawn to certain things, developing an improvement. Gary, I’ve taken up way too much of your time this far. I can go down hours or rabbit holes with you. I’m going to round this out with two questions here. I see you’re sitting in front of two big bookshelves. People are always interested in what the guests are reading. I’m wondering if there’s any books that you’ve picked up, doesn’t have to do with decision-making or anything like that. Just books throughout your life that you’ve really enjoyed.

[01:16:55] Gary: You should have asked me that question in advance so that I could think about it. That’s a really tough question. What are the books that changed me? Okay. One book that really changed me was when I was a graduate student and it was a book called Cognitive Psychology by Ulric Neisser  And it was published in 1967. And I read that book and I was just knocked out and I said, even though my department at the University of Pittsburgh didn’t have anybody in the area of cognitive psychology, it was an area that was just emerging. I said, this is what I want to do. And so that’s a book that had a tremendous influence on me. 

Another book that had a tremendous influence was a book by Hubert Dreyfus, What Computers Still Can’t Do which was a critique of artificial intelligence. Just made me think more about how people function and how they’re not just robots being robots, following rules. So that’d be a second one. And a third, I think is Thought And Choice In Chess. Where he did research with chess grandmasters. He was also I think he was a chess master elf and he just studied chess, grandmasters games, handling chess problems, thinking aloud, seeing what they were up to and really doing cognitive interviewing.

And it was amazing to see what he was able to do. So those are three books that stand out for me over my career. The book that I’m reading now that I’m finding very fascinating is a book called Unmaking the West. It’s an edited book. My friend, Phil Tetlock, and Richard Ned Lebow and Geoffrey Parker are the editors. It’s about asking various historians and analysts, if you could change one thing, what would you change that would prevent the west from being as dominant as it is, and just to watch their minds at work, trying to do counterfactual reasoning is a treat.

Influential Person Gary Would Interview

[01:19:30] Sean: I’ll have to explore that. Yeah, we just did a deep dive on Phil Tetlock’s book, Superforecasting, all about revisiting decision making. So I love these book recommendations. Final one, then we’ll link the listeners up with everything that you’ve got going on. But if you could do that, just have like one of those deep dive conversations for hours with anyone dead or alive. It’s not a family member or friend who would you love to just be able to sit down and interview?

[01:20:06] Gary: Who would I interview? The guy jumping to the front of my mind is Lyndon Johnson. He was not an admirable man and but having read the volumes of Robert Caro’s biography of Lyndon Johnson, especially Master of the Senate and to try to understand how he was able to take such a totally almost dysfunctional entity and bended to his wealth, how he was able to figure out how to get people to do what he wanted.

People who were sworn opponents of him, those southerners, who were opposed to desegregation orders, signing on, the things he accomplished were so amazing. He’s the guy I would love. I mean, he’s the one who stands up. There’s one more volume of Caro’s biography, but it hasn’t been published yet. And all Caro’s fans are sort of waiting for that last volume to get published. We can’t wait.

[01:21:40] Sean: I love it. That’s the first time he’s come up as an answer, but Gary, this has been so intriguing. Your work’s really impacted me. I want to make sure we link you up, a workup for the listeners where we should direct them? We’ll have everything linked up in the show notes. They can easily click, but anywhere you want them checking out?

[01:21:56] Gary: The easiest place is to go to my company’s website, www.shadowboxtraining.com.

[01:22:08] Sean: Fantastic. I’ll also have your books linked up and then some of the great papers you’ve been part of. But Gary Klein, I cannot thank you enough for joining us on What Got You There.

[01:22:16] Gary: Okay, thank you very much. I really enjoyed this conversation. This was very, very delightful. You had one tough, fascinating question after another that’s a real workout.

[01:22:33] Sean: You guys made it to the end of another episode of What Got You There. I hope you guys enjoyed it. I really do appreciate you taking the time to listen all the way through. If you find value in this, the best way you can support the show is giving us a review, rating it, sharing it with your friends and also sharing on social media. I can’t tell you how much I appreciate it. Looking forward to you guys, listening to another episode.

.