WASHINGTON—What makes someone a good predictor of upcoming events? The U.S. intelligence community has run several projects aimed at answering that question. That includes a tournament called the Aggregative Contingent Estimation.
Canadian-born Philip Tetlock’s team dominated the four-year contest. In fact, it was so dominant the competition ended, and only his team remained. The U.S. government has now released data from his team, to help future researchers.
Tetlock offers tips for others hoping to hone their skills as forecasters. He lists Ten Commandments for forecasters in the book he co-authored with Dan Gardner: “Superforecasting: The Art and Science of Prediction.”
1) Triage: Ask the right questions. Be selective. Aim for so-called Goldilocks questions — not too easy, not too hard. For example, asking who will win the 2028 U.S. presidential election is pointless. Pick achievable targets.
2) Break it up: Tackle your question piece by piece. A forecaster in London, Peter Backus, wondered how many potential female partners there might be for him in the city. Rather than making a blind guess, he approached the challenge in chunks. He started with the population of London (approximately six million), divided by two for gender, divided by two once again for the single population, guesstimated that about 20 per cent were in his ideal age range, 26 per cent were university graduates, figured five per cent might find him attractive, he might find five per cent attractive, and supposed about 10 per cent would be compatible with him. Conclusion: 26 potential mates in London.
3) Balance inside and outside views: Seek evidence of a precedent, and incorporate past examples into your estimate.
4) Strike the right balance between under- and overreacting to details: Don’t get overly excited by one single development, but do recognize whether it warrants an upgrade to your outlook. “The best forecasters tend to be incremental belief updaters, often moving from probabilities of, say, 0.4 to 0.35 or from 0.6 to 0.65.”
5) Push back against your biases: For example, “If you are a devout dove who believes that threatening military action never brings peace, be open to the possibility that you might be wrong about Iran. And the same advice applies if you are a devout hawk who believes that soft ‘appeasement’ policies never pay off.”
6) Strive to distinguish as many degrees of doubt as the problem permits: A good poker player knows the difference between a 60/40 bet, a 40/60, and 45/55. It’s the same with forecasting. Don’t settle for lazy predictions that yes, no, or maybe an event will occur: “Your uncertainty dial needs more than three settings.”
7) Strike the right balance between under- and overconfidence, between prudence and decisiveness: There’s a risk in extremes. Don’t be either a blowhard, or a waffler.
8) Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases: Conduct a post-mortem after a bad call. What caused it? Try learning from that mistake — but don’t over-learn it, as it could be a one-time event.
9) Bring out the best in others and let others bring out the best in you: Don’t be afraid to consult, and work with others.
10) Master the error-balancing bicycle: Practice forecasting often enough, and work at learning to juggle all these challenges at once.
And the book concludes with an 11th commandnment: “Don’t treat commandments as commandments.”