Are crowds wise?

According to the "wisdom-of-crowds" hypothesis, when people estimate some numerical quantity (such as the number of jelly beans in a jar") the crowd judgment - i.e. the average of all estimates - is more accurate than the majority of individual judgments.  A working paper by Joseph Simmons, Leif Nelson, and their colleagues suggests that the "wisdom of crowds" hypothesis may be over-rated. American football fans who engaged in spread betting not only lost most of the time, but placed worse bets as the season progressed. Another group of fans who engaged in a different, but logically-equivalent,  form of betting did much better.

For an entire season the researchers asked highly-knowledgeable fans of the (American) National Football League to make regular predictions about the outcomes of games. Some fans were asked to place bets against a points spread; for example, with a four-point spread against the Washington Redskins someone placing a bet on that team would only win if the Redskins won by a margin of more than four points. Someone betting on the Redskins' opponents would win if that team won or if the Redskins won by four points or less. Crucially for the purpose of this study, the points spreads used were increased by the researchers in an attempt to reduce or eliminate any advantage to betting on the favourite. This manipulation worked; against the points spread favourites lost more games than they won.

Other fans were asked to place a different, but logically-equivalent bet; they were asked to predict the winning team and to estimate the margin of victory. They received payments that depended on their accuracy. A third group of fans betted using a combination of the other two conditions. They first made a prediction against the points spreads, and then predicted who would win and by what margin.

In their main analyses, Simmons et al took the crowd judgment to be those cases where the proportion of money bet on the favourite was more, less, or equal to 50%.  Contrary to the wisdom-of-crowds hypothesis, the crowd put its money on the favourite in 89% of games (thus lost money most of the time) and the crowd was worse than 93% of its members. For a subset of fans who had been warned that the spread had been adjusted, the crowd bet on the favourite only slightly less often (82.7% of the time) and still lost more often than they won. When people bet on favourites they placed more money than when they bet on underdogs, which suggests that they believed that the favourite would beat the points spread.

Was it the case that wisdom increased with experience? Did fans bet more appropriately with the passage of time (and experience of feedback)? No - as the season progressed they increasingly placed their money on favourites, thereby increasing their losses.

By contrast, those fans who were asked to predict the winner and estimate the points difference did much better. The crowd bet on the underdog 82.7% of the time, won against the spread 55.4% of the time, and outperformed 95.6% of its members. There was no change in performance over time.

The third group of fans, those who bet against the spread and predicted the victor/margin of victory, did just a little worse than those who only did the latter, but showed no overall bias towards betting on the favourite (unlike those who only bet against the spread).

Simmons et al explain these results in terms of their theory of intuitive confidence (Simmons and Nelson, 2006; see Chapter 15 in Hardman, 2009). They suggest that fans' intuitions about who will win leads them to underweight the points spread that they are presented with. However, the biasing influence of their intuition is attenuated when they themselves are required to think about what the points spread should be.

The overall message of the paper is that the extent to which crowds may or may not be wise depends on the way in which individuals are required to make their judgments.


This research was reported at the 2009 meeting of the Society for Judgment and Decision Making, Boston, USA. The working paper is:
Simmons, J.P., Nelson, L.D., Galak, J., and Frederick, S. (April 2009). Are crowds wise when predicting against points spreads? It depends on how you ask.


References

Simmons, J.P., and Nelson, L.D. (2006). Intuitive Confidence: Choosing Between Intuitive and Nonintuitive Alternatives. Journal of Experimental Psychology: General, 135, 409-428.

Under-achievement and the glass ceiling

Report on a presentation to the 2009 conference of the Society for Judgment and Decision Making

Saturday 21st November.

Under-achievement and the glass ceiling (Robin Hogarth and Natalia Karelaia - presented by Natalia Karelaia)

A question that is frequently asked about the world of work is "Why are there so few women at the top"? Answers that have been suggested focus on discrimination, child rearing preferences, and preferences relating to competition. One difficulty with studying the workplace is that there may be multiple factors that are hard to disentangle.

In this conference presentation, Natalia Karelaia reported research into men and women's behaviour in a television game show. The gameshow studied was the first season of a Columbian programme, El Jugador. The show involves regular games, Sunday games, and a Final. In each game six contestants take part in 5 rounds of general knowledge questions, where 75$ to 375$ can be won for each correct answer. Contestants are ranked according to their accumulated gains, although they themselves are not told how they are ranked; rather, they only receive feedback on their gains. After each round, players can exit the game if they wish. If nobody exits, then the lowest-ranking contestant is expelled and loses all accumulated prize money. Before making their decision about whether or not to withdraw contestants can interact publically to gain strategic advantage (e.g. by bluffing).

Hogarth and Karelaia studied 36 regular games and 6 Sunday games. These involved a total of 216 players, 47% of whom were women. During the course of these games 90 players were expelled and 125 left voluntarily. The mean payoff was 2618$ and the median was 625$ (hence a skewed distribution). The winner of the final received a payoff of 29,625$. Across the games, the women earned 50-60% less than the men, and as games progressed the proportion of women decreased. So a question of central interest is: Are the women expelled or do they exit themselves? In regular games, more women than men were expelled, with women achieving 51% accuracy compared to 56% for men. However, there were also more voluntary withdrawals for women, so did these women withdraw because of a lack of skill or for some other reason? Hogarth and Karelaia found that whatever the size of women's accumulated gains, they had a greater probability of exiting the game.

Furthermore, an analysis of premature exits - those departures from the game by people who were not ranked last -  showed that the probability of withdrawal was 5 % more for women. In terms of correct exits - i.e. the lowest-ranked people voluntarily withdrawing - there was no difference for women and men. However, it was also the case that the probability of women prematurely withdrawing increased as the number of women in the competition declined.

Other analyses showed that men more often engaged in strategic bantering (to persuade others to quit) and that female winners-to-be were as active in the interactions of the regular games as were the male contestants.

One implication of these results is that women in competitive environments may need more social support (ie interactions with others). In particular, because there is a cumulative effect of female withdrawals on the likelihood of further withdrawals, it may be that workplace environments need to maintain a substantial proportion of women at lower levels in order to ensure that women are properly represented at the top.

Needless to say, although this study manages to examine men and women's behaviour without many of the complicating factors that may be involved in workplace settings, this also means that care must be taken not to make overly-strong generalizations. This is also a naturalistic study rather than a true experiment, with the limitations that that brings. These and other issues can be found in this working paper version of the study. Qualification: At the time of writing I don't know whether this work has yet been submitted for peer review.

Humans prefer cockiness to expertise - life - 10 June 2009 - New Scientist

Humans prefer cockiness to expertise - life - 10 June 2009 - New Scientist

Shared via AddThis