Monday, 30 November 2009

Are crowds wise?

According to the "wisdom-of-crowds" hypothesis, when people estimate some numerical quantity (such as the number of jelly beans in a jar") the crowd judgment - i.e. the average of all estimates - is more accurate than the majority of individual judgments.  A working paper by Joseph Simmons, Leif Nelson, and their colleagues suggests that the "wisdom of crowds" hypothesis may be over-rated. American football fans who engaged in spread betting not only lost most of the time, but placed worse bets as the season progressed. Another group of fans who engaged in a different, but logically-equivalent,  form of betting did much better.

For an entire season the researchers asked highly-knowledgeable fans of the (American) National Football League to make regular predictions about the outcomes of games. Some fans were asked to place bets against a points spread; for example, with a four-point spread against the Washington Redskins someone placing a bet on that team would only win if the Redskins won by a margin of more than four points. Someone betting on the Redskins' opponents would win if that team won or if the Redskins won by four points or less. Crucially for the purpose of this study, the points spreads used were increased by the researchers in an attempt to reduce or eliminate any advantage to betting on the favourite. This manipulation worked; against the points spread favourites lost more games than they won.

Other fans were asked to place a different, but logically-equivalent bet; they were asked to predict the winning team and to estimate the margin of victory. They received payments that depended on their accuracy. A third group of fans betted using a combination of the other two conditions. They first made a prediction against the points spreads, and then predicted who would win and by what margin.

In their main analyses, Simmons et al took the crowd judgment to be those cases where the proportion of money bet on the favourite was more, less, or equal to 50%.  Contrary to the wisdom-of-crowds hypothesis, the crowd put its money on the favourite in 89% of games (thus lost money most of the time) and the crowd was worse than 93% of its members. For a subset of fans who had been warned that the spread had been adjusted, the crowd bet on the favourite only slightly less often (82.7% of the time) and still lost more often than they won. When people bet on favourites they placed more money than when they bet on underdogs, which suggests that they believed that the favourite would beat the points spread.

Was it the case that wisdom increased with experience? Did fans bet more appropriately with the passage of time (and experience of feedback)? No - as the season progressed they increasingly placed their money on favourites, thereby increasing their losses.

By contrast, those fans who were asked to predict the winner and estimate the points difference did much better. The crowd bet on the underdog 82.7% of the time, won against the spread 55.4% of the time, and outperformed 95.6% of its members. There was no change in performance over time.

The third group of fans, those who bet against the spread and predicted the victor/margin of victory, did just a little worse than those who only did the latter, but showed no overall bias towards betting on the favourite (unlike those who only bet against the spread).

Simmons et al explain these results in terms of their theory of intuitive confidence (Simmons and Nelson, 2006; see Chapter 15 in Hardman, 2009). They suggest that fans' intuitions about who will win leads them to underweight the points spread that they are presented with. However, the biasing influence of their intuition is attenuated when they themselves are required to think about what the points spread should be.

The overall message of the paper is that the extent to which crowds may or may not be wise depends on the way in which individuals are required to make their judgments.

This research was reported at the 2009 meeting of the Society for Judgment and Decision Making, Boston, USA. The working paper is:
Simmons, J.P., Nelson, L.D., Galak, J., and Frederick, S. (April 2009). Are crowds wise when predicting against points spreads? It depends on how you ask.


Simmons, J.P., and Nelson, L.D. (2006). Intuitive Confidence: Choosing Between Intuitive and Nonintuitive Alternatives. Journal of Experimental Psychology: General, 135, 409-428.

No comments:

About Google Analytics

This website uses Google Analytics, a web analytics service provided by Google, Inc. (“Google”). Google Analytics uses “cookies”, which are text files placed on your computer, to help the website analyze how users use the site. The information generated by the cookie about your use of the website (including your IP address) will be transmitted to and stored by Google on servers in the United States . Google will use this information for the purpose of evaluating your use of the website, compiling reports on website activity for website operators and providing other services relating to website activity and internet usage. Google may also transfer this information to third parties where required to do so by law, or where such third parties process the information on Google's behalf. Google will not associate your IP address with any other data held by Google. You may refuse the use of cookies by selecting the appropriate settings on your browser, however please note that if you do this you may not be able to use the full functionality of this website. By using this website, you consent to the processing of data about you by Google in the manner and for the purposes set out above.