Harnessing Crowd Wisdom

If three students were asked to estimate the price of a burrito at a popular local chain whose true price was 10, they might answer 9, 10, and 11, respectively. The classic “wisdom of crowds” finding is that the mean of their responses is very accurate even though not every response is very accurate (see Surowiecki's book). This phenomenon is robust but not always as effective as this example. If the same three students were asked about the exchange rate between South African rands and US dollars (also about 10 when I wrote this in 2013), for example, they might give the answers 1, 10, and 100. The mean of their responses (37) is far from the truth; it seems that they are not as wise about the rand as they are about the burrito, perhaps because of greater uncertainty. Below I argue that this crowd is not necessarily unwise about the exchange rate, but rather is wise in a different way.

The key insight is that different levels of certainty call for different types of averaging.  The standard mean used in mathematics and statistics is technically called the arithmetic mean. Another important but very uncommon mean is the geometric mean. Though the arithmetic mean of the students’ exchange rate estimations is 37, far from the truth, their geometric mean is 10, exactly right. The geometric mean is defined as the product of all n responses, raised to the power of 1/n. It is equivalent to taking the average of the logarithm of all responses, and exponentiating this average.

The difference between arithmetic and geometric means corresponds to different ways for crowds to be wise. Arithmetic means cancel out symmetric errors in unaltered values. If crowds have high certainty or high expertise, and can agree on the order of magnitude of the parameter to be estimated, then arithmetic means yield good results: these crowds are wise about values. Geometric means, by contrast, cancel out symmetric errors in orders of magnitude. Crowds dealing with great uncertainty are usually not wise about values, but they can be wise about orders of magnitude. Using a geometric mean is essentially equivalent to asking students “how many zeroes does this exchange rate have?” and taking an arithmetic mean of responses. It is a different, and (we will argue) more effective method of eliciting crowd wisdom under uncertainty.

I predict that geometric means can systematically outperform arithmetic means in some estimations. In other words, I predict that a crowd that is not wise about values can be wise about orders of magnitude. I further predict that geometric means will tend to be more accurate than arithmetic means in situations of high uncertainty.

Crowd wisdom is there, but it's not always straightforward to access it. Deciding exactly which mean to examine is one important consideration for those who wish to harness the wisdom of the crowd.

 

Post by Bradford Tuckfield

I offer a full range of data analysis services. To set up a free consultation, email me at bradford@analytics-man.com.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>