Looking for a Good Forecast?

Michael Giberson

Philip Tetlock – the expert on political expertise – reviews three books on political forecasting for The National Interest, but the problem of selecting a good forecast equally applies in commerce:

Reading these three books, it is easy to feel like a frustrated shopper wandering aimlessly down the forecasting aisle in the supermarket of ideas. The products on offer are packaged well—but we have no objective benchmarks, no trusted Consumer Reports, against which to gauge performance. We have no idea whether we would be better-off paying one of these consultancies gobs of money for their proprietary forecasts or simply downloading the latest odds from a high-profile prediction market that culls individual bets on world events such as Tradesport. Indeed, would we do as well relying on the dart-throwing chimps or mindless extrapolation rules, like “Predict the most recent rate of change”?

Tetlock observes that pundits like to both make bold claims and hedge their bets: bold, because pundits need to stand out from the crowd, and hedge, because pundits can’t afford to be tied to flat-out mistakes.  Tetlock wants to pin forecasters down so we can score their track record.  (I’m in favor, see my post at Midas Oracle: “Separating cheap talk from truly held beliefs.”)

In response to Tetlock, Robin Hanson wonders whether people don’t care so much about the accuracy of forecasts, but really just want to affiliate with high-status impressive folks by reading their provocative forecasts.

In the commercial world, say the oil and gas business, while affiliating with high-status impressive folks is good (why else CERAWeek?), people have a pretty strong incentive to care about the accuracy of forecasts, too.  There are a lot of oil and gas forecasting companies out there, and they make reasonably clear quantitative forecasts and so should be relatively easy to score.  Has anyone produced a “shopper’s guide to oil and gas forecasting companies”?