“How social influence can undermine the wisdom of crowd effect”

Michael Giberson

New in the Proceedings of the National Academy of Sciences, a paper showing how the “wisdom of crowd” effect can be undermined when members of the “crowd” are given information about what the rest of the crowd predicted. Averaged or full information about others’ predictions tended to narrow the diversity of predictions in subsequent trials without improving the accuracy of the average prediction.

See a longer discussion “Sharing Information Corrupts Wisdom of Crowds” at the WIRED Science blog. HT to Mark Thoma, Economist’s View.

Citation: Jan Lorenz, Heiko Rauhut, Frank Schweitzer, and Dirk Helbing, 2011, “How social influence can undermine the wisdom of crowd effect,” PNAS, Doi: 10.1073/pnas.1008636108.

Abstract: Social groups can be remarkably smart and knowledgeable when their averaged judgements are compared with the judgements of individuals. Already Galton [Galton F (1907) Nature 75:7] found evidence that the median estimate of a group can be more accurate than estimates of experts. This wisdom of crowd effect was recently supported by examples from stock markets, political elections, and quiz shows [Surowiecki J (2004) The Wisdom of Crowds]. In contrast, we demonstrate by experimental evidence (N = 144) that even mild social influence can undermine the wisdom of crowd effect in simple estimation tasks. In the experiment, subjects could reconsider their response to factual questions after having received average or full information of the responses of other subjects. We compare subjects’ convergence of estimates and improvements in accuracy over five consecutive estimation periods with a control condition, in which no information about others’ responses was provided. Although groups are initially “wise,” knowledge about estimates of others narrows the diversity of opinions to such an extent that it undermines the wisdom of crowd effect in three different ways. The “social influence effect” diminishes the diversity of the crowd without improvements of its collective error. The “range reduction effect” moves the position of the truth to peripheral regions of the range of estimates so that the crowd becomes less reliable in providing expertise for external observers. The “confidence effect” boosts individuals’ confidence after convergence of their estimates despite lack of improved accuracy. Examples of the revealed mechanism range from misled elites to the recent global financial crisis.

Julian Simon and John Tierney were just lucky, he guesses

Michael Giberson

Cornucopian views on resources are back in the news a bit due to John Tierney’s column at the New York Times in which he reports the settling of a wager on oil prices entered into in 2005 with oil industry analyst/peak oil proponent Matt Simmons. In brief, Simmons bet Tierney oil prices would average $200 bbl (in 2005$) during 2010, oil prices were substantially lower than that in 2010, and Tierney won the bet.

But of course that bet, and the results of the more famous related bet between Julian Simon and Paul Erhlich, don’t settle arguments. Money may change hands, but there is no end to disputation.

At The Oil Drum, David Murphy writes: “Lucky Economists, Unlucky Scientists?”  Murphy makes a number of useful points as he stumbles toward some sort of conclusion that will help him protect his preexisting pessimistic views.  Curiously, the conclusion he comes to is that Ehrlich and Simmons and Simon and Tierney must all be foolish idiots, or at least that is the implication:

The bets made by Ehrlich and Simon as well as Simmons and Tierney were faulty because they assumed ceteris paribus conditions; that all other conditions aside from the one on which the bet is made (depletion in these cases) will not influence prices. In the real world, however, there are a number of factors that influence price. As a result, it is incorrect for Tierney to claim that his victory, or that of Simon, is a validation of the economists’ viewpoint on the price of commodities. The economists were lucky, and the scientists unlucky.

Murphy wants us to believe that these four gentlemen assumed that nothing else in the world would change over the five (Simmons-Tierney) or ten (Simon-Ehrlich) years of the wagers? C’mon, these aren’t a bunch of schoolboys betting their lunch money. Somehow I think we need to give them a little more credit than that.

Sure, two guys settling a public bet doesn’t establish what understanding of the nature of the world and society is most reflective of reality. But claiming that folks with opposing views are “just lucky” doesn’t seem like a useful way to advance understanding of the world, either.

When will manipulation of public prediction markets begin to work?

Michael Giberson

At Constructive Economics, Abe Othman discusses a purported manipulation attempt in Intrade’s Health Care Reform bill market.  The nut of the story is that early on March 17th a trader apparently poured a bit of money into the market, briefly driving the price from around 60 down to 35.  After a few hours the price bounced back into the 60s; if it was manipulation, it failed.  But Othman speculates about a future in which manipulation would work.

Because the Intrade price can be interpreted as an estimate of the likelihood that the bill will pass, a sharp fall in the price could indicate new information reaching the market suggesting the bill will fail.  In the manipulation story presented by Othman, a new perception the bill is failing could be used to pressure the weakest members of the coalition supporting the bill to drop out (Maybe the argument goes, “Why go down with a sinking ship, when you and your constituents never wanted the ship in the first place?”).  As support actually falls, the likelihood the bill passes drops with it.  The manipulated price becomes, with a little lobbying, a correct prediction.

While Othman recognizes that the purported manipulation failed this time, he wonders whether prediction market prices will become sufficiently trusted that such a manipulation will work.  In fact, he predicts, “It’s only a matter of time, a couple years I would guess, before the kind of manipulation I’ve described actually works.

I disagree.

While it is true that a trader can often move the Intrade price relatively cheaply, because the markets often are thin, it is well known that a trader can move the Intrade price.  No half-way sophisticated interpreter of Intrade price data would take a sudden sharp move based on a few trades as proof of changing fundamentals, at most it might inspire the viewer to scan for new news.  It was only a few hours after the March 17 episode before bloggers were calling “manipulation!”  Are observers going to become less willing to call “manipulation!” in a couple of years? No.

While it is true that a trader can often move the Intrade price relatively cheaply, because the markets often are thin, holding the market to the manipulated target price can get expensive.  A manipulator can’t buy the price signal, he just rents it for a while.  And the rental rate will tend to rise over time because the mis-pricing will attract informed traders to trade against the manipulator.

Maybe this gets interesting.  So long as the markets are thinly traded then the market signal can be rented cheaply, but observers treat the signal as cheap talk.  What if talk is not cheap?  Can a deep pockets manipulator actually buy the market price?  That is to say, can the manipulator rent the signal long enough to overcome the “cheap talk” dismissal and change the likelihood of the outcome? I’d say this would work only in a world in which enough market observers  trust the market price summary more than all of the other information available about the subject of the prediction market, but this is unlikely to be the world we live in.

I predict: this kind of manipulation will not happen within the next several years.

Onions and motion picture box office receipts

Michael Giberson

Felix Salmon had an op-ed in New York Times on Hollywood’s opposition to the trading of future’s contracts based on box office receipts.  Salmon said:

In the 1950s, onion growers were often shocked at the low prices they were getting. Casting around for a villain to blame, they alighted on derivatives traders, and they persuaded Congress to ban any futures trading in onions.

Today onions are the only commodity for which futures trading is banned. Not coincidentally, onion prices remain extremely volatile: they doubled in 2008, and then fell by 25 percent in 2009.

Today, no one is silly enough to ask a member of Congress to simply outlaw futures trading in a certain type of contract — no one, that is, except Hollywood film producers. Under the proposed financial-reform legislation making its way through the Senate, the bit of the 1958 bill saying “except onions” would be amended to read “except onions and motion picture box office receipts.”

Looking back at 2008-2009 commodity prices, we see a number of other goods for which prices were extremely volatile, some of which were traded on futures market.  (Consider natural gas prices, which began 2008 at $7.80 per mmbtu, rose to over $13.50 mid-year, then tumbled as low as $2.51 in September 2009.)  So Salmon’s brief nod to data on volatility leaves much out.  Fortunately, the onion futures trading ban has been much studied, and analysis pretty strongly comes to the conclusion that the onion futures trading ban has increased spot price volatility.

But Salmon’s observation is more to the point that advocates of bans in futures trading don’t always get what they want, even if, as Salmon said, it isn’t very clear why most of Hollywood seems opposed to the markets.

LINK to recent related KP posts.

HT Chris Masse.

Will the CFTC allow the newly approved box office futures exchange to actually offer box office futures contracts?

Michael Giberson

As just noted, the CFTC has approved Media Derivatives request to establish Trend Exchange, a box office futures exchange.  At Midas Oracle, Chris Masse reacts to the news by drawing attention to remarks by CFTC commissioners suggesting that it may be difficult for Trend Exchange to gain the subsequent CFTC approvals needed for it to actually offer box office futures contracts.

From a Bloomberg story on the exchange approval:

The CFTC must still approve the type of contracts to be dealt before Trend Exchange can begin. The company has said its first product will center on opening-weekend box office.

Product approval is “a very different question” from exchange approval and raises “significant concerns,” Chilton said in an e-mailed statement. He said he had “reluctantly” concurred in today’s vote.

“We have serious concerns regarding the trading of media contracts and we support a very thorough review of all of these first-of-a-kind products,” CFTC Commissioner Scott O’Malia said in an e-mailed statement.

U.S. Senator Blanche Lincoln, an Arkansas Democrat, today added language banning trade in movie futures to a broader derivatives bill she is writing. Lincoln is chairman of the Agriculture Committee that oversees the commodity commission.

More information from the CFTC here, including links to the order and related.

While the CFTC’s inquiry regarding event contracts seems relevant here, I see no reference to that proceeding from 2008 in the CFTC order or concurring statements issued by commissioners.  Perhaps, too, that is a bad sign.

CFTC approves box office futures market

Michael Giberson

Today the CFTC approved Media Derivatives Inc.’s request to create a futures exchange based on box office receipts.  The exchange “is primarily focused on the development of a variety of products to benefit the entertainment industry with one if its initially proposed products being designed to help mitigate risk and enhance the successful financing of motion pictures through trading of opening weekend domestic box office receipts.”

See also reports at Wall Street Journal and Los Angeles Times.

Media Derivatives’s Trend Exchange market is one of two similar proposals that have been submitted to the CFTC for approval.  The other proposal has been submitted by Cantor Fitzgerald, a Wall Street investment and brokerage company, which acquired play-money site Hollywood Stock Exchange a few years ago.

ADDENDUM: Will the CFTC allow the newly approved box office futures exchange to actually offer box office futures contracts?

Prediction markets for movie box office: Manipulation! Insider Trading! Efficiency! and Twitter!

Michael Giberson

Tyler Cowen links to a story in the New York Times detailing the opposition of the Motion Picture Association of America to two proposed markets for forecasting movie ticket sales.  Of the objections noted in the article, Cowen thinks the key issue for Hollywood is the possibility of a poor market showing making it difficult for a film to get into and stay in theaters.  The article also mentions possible market manipulation and insider trading issues as problems.

Robin Hanson addressed the manipulation arguments a few days ago, “Movie Manipulation“:

My research suggests that speculative markets are remarkably robust to manipulation attempts; the more folks try to manipulate, the more accurate market estimates get on average!  But with limited funding, I’ve only done a limited number of experiments; I can’t prove no one will ever use a speculative market to purposely influence movie perceptions.  And alas this mere possibility of manipulation may seem intolerable.

Part of what the movie industry fears is further loss of influence over pre-release product positioning, but this is clearly an area where “information wants to be free” will overcome the industry’s desire to control the pre-release buzz.

Rumors and reports now spread more efficiently, thanks in part to the internet, and once a film is out the internet gives us “word of mouth on steriods.”  Fast Company reports: “Two researchers at HP Labs, Sitaram Asur and Bernardo Huberman, have discovered that you can actually use Twitter mentions to predict how well a movie will do in it’s first couple weekends of release.”

The Fast Company article notes that Twitter analysis method actually outpredicts the (play money) movie prediction market Hollywood Stock Exchange.  At Midas Oracle, Chris Masse countered, “You could turn Bernardo Huberman’s study around and say that the HSX traders are not yet using Twitter as a source to the full extent possible.”  Now the Twitter angle is public information, that omission will be overcome. (And how soon before Hollywood studios begin mass-Twitter marketing campaigns?)

See also: Deadline,MPAA Organizes Entertainment Community Opposition To Movie Futures Exchange.”

Did prediction markets miss the call on Chicago’s Olympic bid?

Michael Giberson

The IOC recently selected Rio de Janerio over three competing bids to host the 2016 summer Olympic games.  The Chicago bid was favored in public prediction markets, with prices at Intrade between 50 and 60 at the time of decision and prices at Betfair implying about a 50 percent chance.  Did the prediction markets fail to predict well?

At Midas Oracle, Chris Masse has been asserting that prediction markets for IOC selections are fundamentally flawed, saying that the IOC is a small, secretive committee that doesn’t leak information and therefore no information is “out there” available to be aggregated by a prediction market. He was saying this before the IOC vote, too; this is not just after-the-fact speculation, it was his before-the-fact speculation. (Also posts here, here, here, here, here, here and, from April 2007, this post.)

I think the “small, secretive committee” explanation is weak, so I’ve been poking back a little in the comments. Chris, as is his style, has been elevating my comments into new posts in order to re-assert his views.

But a more fundamental question is whether or not it can be said that the prediction markets got it wrong.  At Sabernomics, J.C. Bradbury reports watching Intrade closely the morning of the IOC decision:

Around 9 AM … the odds show Chicago to be the favorite with a 53% chance of winning, closely followed by Rio at 46%, Tokyo at 3%, and Madrid at 2%. Like all the pundits following the selection were saying, it was a race between Chicago and Rio, but was very close to call. These odds also show something else, Chicago was trending down and Rio was trending up. The trend would continue for the next few hours.

… Looks like useful information was leaking out from knowledgeable parties just before the vote. This is evidence for, not against, the strong-form of efficient markets hypothesis.

Bradbury does an excellent job sifting through the shifting coalitions revealed in the three rounds of IOC voting.  Neither Madrid nor Toyko showed any significant ability to attract votes as the rounds proceeded.  It was going to be Rio or Chicago all along, but Chicago was weakest in the four-way vote and lost early, leaving the games to go to Brazil.

Based on Bradbury’s analyis, I’m convinced that the decision was pretty much a toss up between Chicago and Rio.  That conclusion was also implied in the prediction market prices just before the decision.  Sure, the prediction markets favored Chicago, slightly, over Rio; I don’t think you can call it a miss given the closeness of the decision.

[Related: Market Design and Marginal Revolution both have brief notes; Infectious Greed provides related discussion.]

UPDATE: Chris Masse doesn’t like my analysis: Who has the best analysis for Chicago’s failed bid for the Olympics?; neither does Paul Hewitt: “Michael Giberson is wrong to imply that the prediction was accurate on the basis that Chicago and Rio were fairly close.” See also here.

Does “putting your money where your mouth is” make betting a free speech issue?

Michael Giberson

Miriam Cherry and Robert Rogers explore the interaction of “Prediction Markets and the First Amendment.”  If prediction markets are “expressive,” does that mean that U.S. government actions that constrain prediction market development potentially raise First Amendment issues on free speech grounds? The authors propose a way forward in which courts, at least until the legislative and administrative branches clarify policy, apply a Miller v. California-like test for identifying permissible prediction markets.

Miller v. California established a three-part test for regulating obscenity, and it is the third part of that test is key for Cherry and Rogers: “whether the work, taken as a whole, lacks serious literary, artistic, political, or scientific value.”  On these grounds Cherry and Rogers suggest the first amendment may offer protection for prediction markets directed at political, economic, cultural, or scientific topics, but perhaps not for prediction markets for sporting or entertainment events.

I’m not sure First Amendment law is the best protection for prediction markets, but it might be better than nothing.

HT to Chris Masse at Midas Oracle.

Why is Chris Masse making a big deal about a little article on prediction markets in The Economist?

At the prediction markets blog, Midas Oracle, Chris Masse has posted several times (here, here, and don’t miss the remarks in the comments) about the recent piece on prediction markets in The Economist.  Among his recent grand pronouncements:

If you are a prediction market consultant, and have nothing to say about the negative piece from The Economist, then you don’t matter anymore.


What is it that The Economist said that prediction market consultants should have something to say about?

I read article when it showed up online, and then again when the print version arrived and after Chris mentioned it at Midas Oracle, and now a third time since I wonder why he is making a big deal of it.

Fortunately, it is a light read – reading it three times has not been particularly taxing. But there is not a lot of depth there to engage: An intro paragraph, some explanation, three brief examples-two of which illustrate implementation problems, and then the article concludes with a quip that “Perhaps the best way to find out when prediction markets will finally take off is to ask your employees–using a prediction market.”

That’s it.

No searing indictment of the prediction market software vendors and consulting business, no challenge to the theoretical foundations of prediction markets, no rejection of the “wisdom of the crowds.” All the article does is observe that the tool has “yet to take off.”

So my question to Chris remains, “What is it that The Economist said that prediction market consultants should have something to say about?”

(Cross-posted at Midas Oracle.)