Superforecasting: The Art and Science of Prediction

Tetlock, Philip E.; Gardner, Dan

Excellent Mauboussin Research on Superforecasters

Pair with other Decision Making

Learnings from the CIA

How To Decide– Annie Duke 

Think Again – Adam Grant  

---

Summary

• Philip Tetlock’s study of hundreds of experts making thousands of predictions over two decades found that the average prediction was “little better than guessing.” That’s the bad news. Tetlock, along with his colleagues, participated in a forecasting tournament sponsored by the U.S. intelligence community. That work identified “superforecasters,” people who consistently make superior predictions. That’s the good news.
• The key to superforecasters is how they think. They are actively open minded, intellectually humble, numerate, thoughtful updaters, and hard working.
• Superforecasters achieve better results when they are part of a team. But since there are pros and cons to working in teams, training is essential. Instruction in methods to reduce bias in forecasts improves outcomes. There must be a close link between training and implementation. The best leaders recognize that proper, even bold, action requires good thinking.
• Humans have a deep-seated desire to anticipate the future. The demand for forecasts is met by a supply of seers, pundits, and experts expounding on what will happen next. But we generally don’t measure the quality of predictions, and when it has been done the results are unimpressive.
• Superforecasting shows that prediction may not be so futile after all. The Good Judgment Project, part of a forecasting tournament sponsored by the U.S. intelligence community, revealed that some forecasters are not only good but consistently good. Using the best ideas from psychology and careful measurement, the GJP team has been able to provide essential lessons for anyone in the prediction business. Here are some of the main conclusions:
  • Forecasting skill exists. The researchers found that a small percentage of the forecasting population were much more accurate than average and consistently so. They were able to find the superforecasters because they welcomed a large sample of forecasters, asked questions with time frames of a year or less, and kept track of the responses.
  • Way of thinking is vital. Closer analysis of the superforecasters shows that they are bright, but not extraordinarily so. What distinguishes them from regular forecasters is the way they think. Superforecasters are actively open-minded, nondeterministic, intellectually humble, numerate, thoughtful updaters, and hard working.
  • Teams. When superforecasters interact with one another, their predictions improve. But there are pros and cons to working in teams. The pros include more information and the ability to harness the power of aggregation. The cons are social loafing and the risk of groupthink. Teams that do the best have members who have been given instruction and training on how to work together.
  • Training. Training in methods to de-bias forecasts and to collaborate effectively improves outcomes. Proper training has content that is valuable and processes that can be implemented immediately. Training without implementation is a waste.
  • Leadership. At first blush, the qualities associated with leadership appear antithetical to those of the superforecasters. The way to reconcile the two is to acknowledge that proper, even bold, action requires good thinking. And the best leaders recognize that even the best laid plans need to be constantly revised based on the conditions. 

Composite Portrait of a “Superforecaster” 

•  Those who are actively open-minded seek views that are different than their own and consider them carefully. Tetlock and Gardner suggest that if they had to reduce superforecasting to a bumper sticker, it would read, “Beliefs are hypotheses to be tested, not treasures to be guarded.”

Philosophic Outlook

• Cautious: Nothing is certain
• Humble: Reality is infinitely complex
• Nondeterministic: What happens is not meant to be and does not have to happen

Abilities and Thinking Styles

• Actively open-minded: Beliefs are hypotheses to be tested, not treasures to be protected
• Intelligent and knowledgeable, with a “need for cognition”: Intellectually curious, enjoy puzzles and mental challenges
• Reflective: Introspective and self-critical
• Numerate: Comfortable with numbers

Methods of Forecasting

• Pragmatic: Not wedded to any idea or agenda
• Analytical: Capable of stepping back from the tip-of-your-nose perspective and considering other views
• Dragonfly-eyed: Value diverse views and synthesize them into your own
• Probabilistic: Judge using many grades of maybe
• Thoughtful updaters: When facts change, they change their minds
• Good intuitive psychologists: Aware of the value of checking thinking for cognitive and emotional biases

Work Ethic

• A growth mindset: Believe it’s possible to get better
• Grit: Determined to keep at it however long it takes

4 Drivers of Successful Superforecasters 

• Find the right people. You get a 10-15 percent boost from screening forecasters on fluid intelligence and active open-mindedness.
• Manage interaction. You get a 10-20 percent enhancement by allowing the forecasters to work collaboratively in teams or competitively in prediction markets.
• Train effectively. Cognitive debiasing exercises lift results by 10 percent.
• Overweight elite forecasters or extremize estimates. Results improve by 15-30 percent if you give more weight to better forecasters and make forecasts more extreme to compensate for the conservatism of forecasts.

Super Forecaster’s “Tools” 

• Good Questions: The first step in coming to useful answers is to ask good questions. A good question has a clear outcome within a specified period of time and addresses an issue that is relevant to the world. 
  • Questions must have a set time frame. 
  • Open ended predictions are of limited value.
  • Good questions must be relevant – : “when you read the question after time has passed, you smack your forehead
  • and say, ‘If only I had thought of that before!’”
• Keeping Score: Keeping score is crucial because it provides feedback and therefore an opportunity to learn. Forecasters want to improve in two ways. 
  • One way is called “calibration,” which means that your forecasts line up with the outcomes. For example, you are well calibrated if you say that certain events will occur with a 40 percent probability and they actually happen 40 percent of the time.
  • Another way to improve is what the authors call “resolution.” Resolution means that when you are sure something is not going to happen, it doesn’t happen, or when you’re sure it will happen, it does. It’s a measure of conviction. Good calibration and resolution are correlated, but they are distinct. You want to sharpen your skills in both ways.
  •  Use specific probabilities and time horizons whenever possible and to keep track of those forecasts. Numerical probabilities dismiss the risk of misinterpretation or misunderstanding. 
• Approaches to Forecasts 
  • Question Triage: Superforecasters sort questions by difficulty and what can be answered and “effort pays off the most.”  
    • Back of napkin math to quickly calculate probabilities
    • Question Clustering- clustering small questions you can answer, each of which update and provide insight into the bigger question you’re trying to answer. Metaphor to remember- the painting technique of pointillism where you consistently add dots to a painting, no dot by itself means much but together it’s a picture. 
• Leadership
  • Leaders must think and change their course of action as necessary. Orders could be questioned, even criticized, if there was a better way. The message was to be actively open-minded.
  • “The art of leadership consists of the timely recognition of circumstances and of the moment when a new decision is required.”
  • Humility should make a leader think carefully about what he or she is doing, and confidence should give an individual the strength to act. 
  • Ultimately, we want our leaders to be confident, decisive, and visionary. But the lesson of Superforecasting is that thinking is an important precursor to action. Further, circumstances change and it is vital to update views and change strategic course appropriately. 

1. An Optimistic Skeptic

• How predictable something is depends on what we are trying to predict, how far into the future, and under what circumstances.
• The consumers of forecasting—governments, business, and the public—don’t demand evidence of accuracy. So there is no measurement. Which means no revision. And without revision, there can be no improvement.
• “I have been struck by how important measurement is to improving the human condition,” Bill Gates wrote. “You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal…. This may seem basic, but it is amazing how often it is not done and how hard it is to get right.”

• Foresight isn’t a mysterious gift bestowed at birth. It is the product of particular ways of thinking, of gathering information, of updating beliefs.
• Superforecasting demands thinking that is open-minded, careful, curious, and—above all—self-critical. It also demands focus. The kind of thinking that produces superior judgment does not come effortlessly. Only the determined can deliver it reasonably consistently, which is why our analyses have consistently found commitment to self-improvement to be the strongest predictor of performance.
• Statistical algorithms beat subjective judgment, and in the handful of studies where they don’t, they usually tie. Given that algorithms are quick and cheap, unlike subjective judgment, a tie supports using the algorithm. The point is now indisputable: when you have a well-validated statistical algorithm, use it.

2. Illusions of Knowledge

• It’s human nature. We have all been too quick to make up our minds and too slow to change them. And if we don’t examine how we make these mistakes, we will keep making them. This stagnation can go on for years. Or a lifetime. It can even last centuries, as the long and wretched history of medicine illustrates.
• What medicine lacked was doubt. “Doubt is not a fearful thing,” Feynman observed, “but a thing of very great value.” It’s what propels science forward.
• When the scientist tells you he does not know the answer, he is an ignorant man. When he tells you he has a hunch about how it is going to work, he is uncertain about it. When he is pretty sure of how it is going to work, and he tells you, “This is the way it’s going to work, I’ll bet,” he still is in some doubt. And it is of paramount importance, in order to make progress, that we recognize this ignorance and this doubt. Because we have the doubt, we then propose looking in new directions for new ideas. The rate of the development of science is not the rate at which you make observations alone but, much more important, the rate at which you create new things to test. 
• Our natural inclination is to grab on to the first plausible explanation and happily gather supportive evidence without checking its reliability. That is what psychologists call confirmation bias. We rarely seek out evidence that undercuts our first explanation, and when that evidence is shoved under our noses we become motivated skeptics—finding reasons, however tenuous, to belittle it or throw it out entirely.
• Attribute substitution, but I call it bait and switch: when faced with a hard question, we often surreptitiously replace it with an easy one.
• Pattern recognition. With training or experience, people can encode patterns deep in their memories in vast number and intricate detail—such as the estimated fifty thousand to one hundred thousand chess positions that top players have in their repertoire.

3. Keeping Score

• Now look at how foxes approach forecasting. They deploy not one analytical idea but many and seek out information not from one source but many. Then they synthesize it all into a single conclusion. In a word, they aggregate. They may be individuals working alone, but what they do is, in principle, no different from what Galton’s crowd did. They integrate perspectives and the information contained within them. The only real difference is that the process occurs within one skull.

4. Superforecasters

• What there is instead is accountability for process: Intelligence analysts are told what they are expected to do when researching, thinking, and judging, and then held accountable to those standards. Did you consider alternative hypotheses? Did you look for contrary evidence? It’s sensible stuff, but the point of making forecasts is not to tick all the boxes on the “how to make forecasts” checklist. It is to foresee what’s coming. To have accountability for process but not accuracy is like ensuring that physicians wash their hands, examine the patient, and consider all the symptoms, but never checking to see whether the treatment works.

5. Supersmart?

• This sounds like detective work because it is—or to be precise, it is detective work as real investigators do it, not the detectives on TV shows. It’s methodical, slow, and demanding. But it works far better than wandering aimlessly in a forest of information.
• Coming up with an outside view, an inside view, and a synthesis of the two isn’t the end. It’s a good beginning. Superforecasters constantly look for other views they can synthesize into their own.
• There are many different ways to obtain new perspectives. What do other forecasters think? What outside and inside views have they come up with? What are experts saying? You can even train yourself to generate different perspectives. When Bill Flack makes a judgment, he often explains his thinking to his teammates, as David Rogg did, and he asks them to critique it. In part, he does that because he hopes they’ll spot flaws and offer their own perspectives. But writing his judgment down is also a way of distancing himself from it, so he can step back and scrutinize it: “It’s an auto-feedback thing,” he says. “Do I agree with this? Are there holes in this? Should I be looking for something else to fill this in? Would I be convinced by this if I were somebody else?”
• That is a very smart move. Researchers have found that merely asking people to assume their initial judgment is wrong, to seriously consider why that might be, and then make another judgment, produces a second estimate which, when combined with the first, improves accuracy almost as much as getting a second estimate from another person. The same effect was produced simply by letting several weeks pass before asking people to make a second estimate. This approach, built on the “wisdom of the crowd” concept, has been called “the crowd within.” The billionaire financier George Soros exemplifies it. A key part of his success, he has often said, is his mental habit of stepping back from himself so he can judge his own thinking and offer a different perspective—to himself.
• There is an even simpler way of getting another perspective on a question: tweak its wording. Imagine a question like “Will the South African government grant the Dalai Lama a visa within six months?” The naive forecaster will go looking for evidence that suggests the Dalai Lama will get his visa while neglecting to look for evidence that suggests he won’t. The more sophisticated forecaster knows about confirmation bias and will seek out evidence that cuts both ways. But if you are constantly thinking the question is “Will he get his visa?” your mental playing field is tilted in one direction and you may unwittingly slide into confirmation bias: “This is South Africa! Black government officials suffered under apartheid. Of course they will give a visa to Tibet’s own Nelson Mandela.” To check that tendency, turn the question on its head and ask, “Will the South African government deny the Dalai Lama for six months?” That tiny wording change encourages you to lean in the opposite direction and look for reasons why it would deny the visa—a desire not to anger its biggest trading partner being a rather big one.
• An element of personality is also likely involved. In personality psychology, one of the “Big Five” traits is “openness to experience,” which has various dimensions, including preference for variety and intellectual curiosity. It’s unmistakable in many superforecasters.
• A brilliant puzzle solver may have the raw material for forecasting, but if he doesn’t also have an appetite for questioning basic, emotionally charged beliefs he will often be at a disadvantage relative to a less intelligent person who has a greater capacity for self-critical thinking. It’s not the raw crunching power you have that matters most. It’s what you do with it.

• For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded.

6. Superquants?

• A smart executive will not expect universal agreement, and will treat its appearance as a warning flag that groupthink has taken hold. An array of judgments is welcome proof that the people around the table are actually thinking for themselves and offering their unique perspectives.
• Epistemic uncertainty is something you don’t know but is, at least in theory, knowable. If you wanted to predict the workings of a mystery machine, skilled engineers could, in theory, pry it open and figure it out.
• Aleatory uncertainty is something you not only don’t know; it is unknowable. No matter how much you want to know whether it will rain in Philadelphia one year from now, no matter how many great meteorologists you consult, you can’t outguess the seasonal averages. You are dealing with an intractably cloud-like problem, with uncertainty that it is impossible, even in theory, to eliminate. Aleatory uncertainty ensures life will always have surprises, regardless of how carefully we plan. Superforecasters grasp this deep truth better than most.
• Charlie Munger sagely observed, “If you don’t get this elementary, but mildly unnatural, mathematics of elementary probability into your repertoire, then you go through a long life like a one-legged man in an ass-kicking contest.”
• For both the superforecasters and the regulars, we also compared individual fate scores with Brier scores and found a significant correlation—meaning the more a forecaster inclined toward it-was-meant-to-happen thinking, the less accurate her forecasts were. Or, put more positively, the more a forecaster embraced probabilistic thinking, the more accurate she was. So finding meaning in events is positively correlated with wellbeing but negatively correlated with foresight. That sets up a depressing possibility: Is misery the price of accuracy?

7. Super news junkies?

• Superforecasting isn’t a paint-by-numbers method but superforecasters often tackle questions in a roughly similar way—one that any of us can follow: Unpack the question into components. Distinguish as sharply as you can between the known and unknown and leave no assumptions unscrutinized. Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomena. Then adopt the inside view that plays up the uniqueness of the problem. Also explore the similarities and differences between your views and those of others—and pay special attention to prediction markets and other methods of extracting wisdom from crowds. Synthesize all these different views into a single vision as acute as that of a dragonfly. Finally, express your judgment as precisely as you can, using a finely grained scale of probability.
• Superforecasters update much more frequently, on average, than regular forecasters. That obviously matters. An updated forecast is likely to be a better-informed forecast and therefore a more accurate forecast. “When the facts change, I change my mind,”
• So there are two dangers a forecaster faces after making the initial call. One is not giving enough weight to new information. That’s underreaction. The other danger is overreacting to new information, seeing it as more meaningful than it is, and adjusting a forecast too radically.
• This suggests that superforecasters may have a surprising advantage: they’re not experts or professionals, so they have little ego invested in each forecast. Except in rare circumstances—

• People base their estimate on what they think is a useful tidbit of information. Then they encounter clearly irrelevant information—meaningless noise—which they indisputably should ignore. But they don’t. They sway in the wind, at the mercy of the next random gust of irrelevant information.
• Superforecasters not only update more often than other forecasters, they update in smaller increments.

8. Perpetual Beta

• To be a top-flight forecaster, a growth mindset is essential. The best illustration is the man who is reputed to have said—but didn’t—“ When the facts change, I change my mind.”
• Keynes was breathtakingly intelligent and energetic, which certainly contributed to his success, but more than that he was an insatiably curious man who loved to collect new ideas—a habit that sometimes required him to change his mind. He did so ungrudgingly. Indeed, he took pride in his willingness to squarely admit mistakes and adopt new beliefs, and he urged others to follow his lead. “There is no harm in being sometimes wrong, especially if one is promptly found out,”

• Try, fail, analyze, adjust, try again:
• To learn from failure, we must know when we fail.

• Once we know the outcome of something, that knowledge skews our perception of what we thought before we knew the outcome: that’s hindsight bias.
• Forecasters who use ambiguous language and rely on flawed memories to retrieve old forecasts don’t get clear feedback, which makes it impossible to learn from experience. They are like basketball players doing free throws in the dark. The only feedback they get are sounds—the clang of the ball hitting metal, the thunk of the ball hitting the backboard, the swish of the ball brushing against the net.

• People often assume that when a decision is followed by a good outcome, the decision was good, which isn’t always true, and can be dangerous if it blinds us to the flaws in our thinking.
• Grit is passionate perseverance of long-term goals, even in the face of frustration and failure. Married with a growth mindset, it is a potent force for personal progress.

9. Superteams

• Success can lead to acclaim that can undermine the habits of mind that produced the success. Such hubris often afflicts highly accomplished individuals. In business circles, it is called CEO disease.

10. The Leader’s Dilemma

• No one ever described Winston Churchill, Steve Jobs, or any other great leader as “humble.” Well, maybe Gandhi. But try to name a second and a third.
• And consider how the superteams operated. They were given guidance on how to form an effective team, but nothing was imposed. No hierarchy, no direction, no formal leadership. These little anarchist cells may work as forums for the endless consideration and reconsideration superforecasters like to engage in but they’re hardly organizations that can pull together and get things done. That takes structure—and a leader in charge.
• “In war, everything is uncertain,” wrote Helmuth von Moltke.
• The most urgent is to never entirely trust your plan. “No plan of operations extends with certainty beyond the first encounter with the enemy’s main strength,” he wrote. That statement was refined and repeated over the decades and today soldiers know it as “no plan survives contact with the enemy.”
• In 1758, when Prussia’s King Frederick the Great battled Russian forces at Zorndorf, he sent a messenger to the youngest Prussian general, Friedrich Wilhelm von Seydlitz, who commanded a cavalry unit. “Attack,” the messenger said. Seydlitz refused. He felt the time wasn’t right and his forces would be wasted. The messenger left but later returned. Again he told Seydlitz the king wanted him to attack. Again Seydlitz refused. A third time the messenger returned and he warned Seydlitz that if he didn’t attack immediately, the king would have his head. “Tell the King that after the battle my head is at his disposal,” Seydlitz responded, “but meanwhile I will make use of it.” Finally, when Seydlitz judged the time right, he attacked and turned the battle in Prussia’s favor. Frederick the Great congratulated his general and let him keep his head.

The fundamental message: think. If necessary, discuss your orders. Even criticize them. And if you absolutely must—and you better have a good reason—disobey them.

• The wise officer knows the battlefield is shrouded in a “fog of uncertainty” but “at least one thing must be certain: one’s own decision. One must adhere to it and not allow oneself to be dissuaded by the enemy’s actions until this has become unavoidably necessary.” 

• Auftragstaktik. Usually translated today as “mission command,” the basic idea is simple. “War cannot be conducted from the green table,” Moltke wrote, using an expression that referred to top commanders at headquarters. “Frequent and rapid decisions can be shaped only on the spot according to estimates of local conditions.” 8 Decision-making power must be pushed down the hierarchy so that those on the ground—the first to encounter surprises on the evolving battlefield—can respond quickly. Of course those on the ground don’t see the bigger picture. If they made strategic decisions the army would lose coherence and become a collection of tiny units, each seeking its own ends. Auftragstaktik blended strategic coherence and decentralized decision making with a simple principle: commanders were to tell subordinates what their goal is but not how to achieve it.
• The battlefield “requires soldiers who can think and act independently, who can make calculated, decisive, and daring use of every situation, and who understand that victory depends on each individual,” the command manual stated.

“Great success requires boldness and daring, but good judgment must take precedence,”

• Eisenhower understood that a cool and assured appearance could do more to spread confidence and boost morale than false claims of certainty. In private, Eisenhower could be a moody, brooding chain-smoker. With the troops, he always had a smile and a steady word.
• How skillfully leaders perform this balancing act determines how successfully their organizations can cultivate superteams that can replicate the balancing act down the chain of command. And this is not something that one isolated leader can do on his own. It requires a wider willingness to hear unwelcome words from others—and the creation of a culture in which people feel comfortable speaking such words. What was done to the young Dwight Eisenhower was a serious mistake, Petraeus says. “You have to preserve and promote the out-of-the-box thinkers, the iconoclasts.”

“Leaders are obligated to respectfully challenge decisions when they disagree, even when doing so is uncomfortable or exhausting. Leaders have conviction and are tenacious. They do not compromise for the sake of social cohesion. Once a decision is determined, they commit wholly.”

• “There is nothing like danger to focus the mind,”

• The humility required for good judgment is not self-doubt—the sense that you are untalented, unintelligent, or unworthy. It is intellectual humility. It is a recognition that reality is profoundly complex, that seeing things clearly is a constant struggle, when it can be done at all, and that human judgment must therefore be riddled with mistakes. This is true for fools and geniuses alike. So it’s quite possible to think highly of yourself and be intellectually humble. In fact, this combination can be wonderfully fruitful. Intellectual humility compels the careful reflection necessary for good judgment; confidence in one’s abilities inspires determined action.
• Engaging in the toughest of all forms of perspective taking: acknowledging that something we despise possesses impressive qualities.

11. Are They Really So Super?

• What makes them so good is less what they are than what they do—the hard work of research, the careful thought and self-criticism, the gathering and synthesizing of other perspectives, the granular judgments and relentless updating.

“It’s funny that a 90% chance of failing people don’t like, but a 10% chance of changing the world people love.”

• If you have to plan for a future beyond the forecasting horizon, plan for surprise.
• Knowing what we don’t know is better than thinking we know what we don’t.

12. What’s Next?

• Effective learning from experience can’t happen without clear feedback, and you can’t have clear feedback unless your forecasts are unambiguous and scorable.

Appendix: Ten Commandments for Aspiring Superforecasters

1. Triage

• Focus on questions where your hard work is likely to pay off. Don’t waste time either on easy “clocklike” questions (where simple rules of thumb can get you close to the right answer) or on impenetrable “cloud-like” questions (where even fancy statistical models can’t beat the dart-throwing chimp). Concentrate on questions in the Goldilocks zone of difficulty, where effort pays off the most.

2. Break seemingly intractable problems into tractable sub-problems.

• Channel the playful but disciplined spirit of Enrico Fermi who—when he wasn’t designing the world’s first atomic reactor—loved ballparking answers to head-scratchers such as “How many extraterrestrial civilizations exist in the universe?” Break apart the problem into its knowable and unknowable parts. Flush ignorance into the open. Expose and examine your assumptions. Dare to be wrong by making your best guesses. Better to discover errors quickly than to hide them behind vague verbiage.

3. Strike the right balance between inside and outside views.

• Superforecasters know that there is nothing new under the sun. Nothing is 100% “unique.” Language purists be damned: uniqueness is a matter of degree. So Superforecasters conduct creative searches for comparison classes even for seemingly unique events, such as the outcome of a hunt for a high-profile terrorist (Joseph Kony) or the standoff between a new socialist government in Athens and Greece’s creditors. Superforecasters are in the habit of posing the outside-view question: How often do things of this sort happen in situations of this sort?

4. Strike the right balance between under- and overreacting to evidence.

• Belief updating is to good forecasting as brushing and flossing are to good dental hygiene. It can be boring, occasionally uncomfortable, but it pays off in the long term. That said, don’t suppose that belief updating is always easy because it sometimes is. Skillful updating requires teasing subtle signals from noisy news flows— all the while resisting the lure of wishful thinking.

5. Look for the clashing causal forces at work in each problem.

• For every good policy argument, there is typically a counterargument that is at least worth acknowledging. For instance, if you are a devout dove who believes that threatening military action never brings peace, be open to the possibility that you might be wrong about Iran. And the same advice applies if you are a devout hawk who believes that soft “appeasement” policies never pay off. Each side should list, in advance, the signs that would nudge them toward the other.

6. Strive to distinguish as many degrees of doubt as the problem permits but no more.

• As in poker, you have an advantage if you are better than your competitors at separating 60/40 bets from 40/60—or 55/45 from 45/55. Translating vague-verbiage hunches into numeric probabilities feels unnatural at first, but it can be done. It just requires patience and practice. The Superforecasters have shown what is possible.

7. Strike the right balance between under- and overconfidence, between prudence and decisiveness.

• Superforecasters understand the risks both of rushing to judgment and of dawdling too long near “maybe.” They routinely manage the trade-off between the need to take decisive stands (who wants to listen to a waffler?) and the need to qualify their stands (who wants to listen to a blowhard?). They realize that long-term accuracy requires getting good scores on both calibration and resolution—which requires moving beyond blame-game ping-pong. It is not enough just to avoid the most recent mistake. They have to find creative ways to tamp down both types of forecasting errors—misses and false alarms—to the degree a fickle world permits such uncontroversial improvements in accuracy.

8. Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases.

• Don’t try to justify or excuse your failures. Own them! Conduct unflinching postmortems: Where exactly did I go wrong? And remember that although the more common error is to learn too little from failure and to overlook flaws in your basic assumptions, it is also possible to learn too much (you may have been basically on the right track but made a minor technical mistake that had big ramifications). Don’t forget to do post mortems on your successes, too. Not all successes imply that your reasoning was right. You may have just lucked out by making offsetting errors.

9. Bring out the best in others and let others bring out the best in you.

• Master the fine art of team management, especially perspective taking (understanding the arguments of the other side so well that you can reproduce them to the other’s satisfaction), precision questioning (helping others to clarify their arguments so they are not misunderstood), and constructive confrontation (learning to disagree without being disagreeable). Wise leaders know how fine the line can be between a helpful suggestion and micromanagerial meddling or between a rigid group and a decisive one or between a scatterbrained group and an open-minded one.

10. Master the error-balancing bicycle.

• Implementing each commandment requires balancing opposing errors. Just as you can’t learn to ride a bicycle by reading a physics textbook, you can’t become a superforecaster by reading training manuals. Learning requires doing, with good feedback that leaves no ambiguity about whether you are succeeding—“I’m rolling along smoothly!”—or whether you are failing—“crash!” Also remember that practice is not just going through the motions of making forecasts, or casually reading the news and tossing out probabilities. Like all other known forms of expertise, superforecasting is the product of deep, deliberative practice.

But don’t forget the “11th commandment”!

“It is impossible to lay down binding rules,” Helmuth von Moltke warned, “because two cases will never be exactly the same.”