We investigated this paradox experimentally, by creating an artificial ‘‘music market’’ in which 14,341 participants downloaded previously unknown songs either with or without knowledge of previous participants’ choices. Increasing the strength of social influence increased both inequality and unpredictability of success. Success was also only partly determined by quality: The best songs rarely did poorly, and the worst rarely did well, but any other result was possible.
Using a ‘‘multiple-worlds’’ experimental design, we are able to isolate the causal effect of an individual-level mechanism on collective social outcomes. We employ this design in a Web-based experiment in which 2,930 participants listened to, rated, and downloaded 48 songs by up-and-coming bands. Surprisingly, despite relatively large differences in the demographics, behavior, and preferences of participants, the experimental results at both the individual and collective levels were similar to those found in Salganik, Dodds, and Watts (2006)...A comparison between Experiments 1 and 2 reveals a different pattern. In these experiments, there was little change at the song level; the correlation between average market rank in the social influence worlds of Experiments 1 and 2 was 0.93.
This is analogous to test-retest error: if you run a media market with the same authors, and same creative works, how often do you get the same results? Forget completely any question about how much popularity correlates with ‘quality’ - does popularity even correlate with itself consistently? If you ran the world several times, how much would the same songs float to the top?
The most relevant rank correlation they seem to report is rho=0.93*. That may seem high, but the more datapoints there are, the higher the necessary correlation soars to give the results you want.
A rho=0.93 implies that if you had a million songs competing in a popularity contest, the #1 popular song in our world would probably be closer to only the #35,000th most popular song in a parallel world’s contest as it regresses to the mean (1000000 - (500000 + (500000 * 0.93))). (As I noted the other day, even in very small samples you need extremely high correlations to guarantee double-maxes or similar properties, once you move beyond means; our intuitions don’t realize just what an extreme demand we make when we assume that, say, J.K. Rowling must be a very popular successful writer in most worlds simply because she’s a billionaire in this world, despite how many millions of people are writing fiction and competing with her. Realistically, she would be a minor but respected author who might or might not’ve finished out her HP series as sales flagged for multi-volume series; sort of like her crime novels published pseudonymously.)
Then toss in the undoubtedly <<1 correlation between popularity and any ‘quality’… It is indeed no surprise that, out of the millions and millions of chefs over time, the best chefs in the world are not the most popular YouTube chefs. Another example of ‘the tails comes apart’ at the extremes and why order statistics is counterintuitive.
* They also report a rho=0.52 from some other experiments, which are arguably now more relevant than the 0.93 estimate. Obviously, if you use 0.52 instead, my point gets much much stronger: then, out of a million, you regress from #1 to #240,000!
A related thing that comes to mind is looking at Hacker News submissions, and how the same post will sometimes get hundreds of upvotes, and other times only get one or two. Example.
Increasing the strength of social influence increased both inequality and unpredictability of success.
Cool. I was going to top-level comment a possible explanation: the more info about other people’s judgements is shared (e.g. via the internet), the stronger misinformation cascades will be, because there are more judgements you observe that are ambiguously either actual data vs. just copying other people’s judgements. With more misinformation cascades, you have more things that are “popular because they’re popular”, and “unpopular because they’re unpopular” (or, stonks and undiscovered good investments).
A different story (or maybe overlapping) is that with more information about people’s judgements, drawing on a larger pool of candidates, the worse the tradeoff becomes to do your own exploration: the cost of mental processing is roughly the same, but just copying everyone else’s opinions gets a much better result (though still maybe significantly worse than is possible).
One of the most interesting media experiments I know of is the Yahoo Media experiments:
“Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market”, Salganik et al 2006:
“Web-Based Experiments for the Study of Collective Social Dynamics in Cultural Markets”, Salganik & Watts 2009:
This is analogous to test-retest error: if you run a media market with the same authors, and same creative works, how often do you get the same results? Forget completely any question about how much popularity correlates with ‘quality’ - does popularity even correlate with itself consistently? If you ran the world several times, how much would the same songs float to the top?
The most relevant rank correlation they seem to report is rho=0.93*. That may seem high, but the more datapoints there are, the higher the necessary correlation soars to give the results you want.
A rho=0.93 implies that if you had a million songs competing in a popularity contest, the #1 popular song in our world would probably be closer to only the #35,000th most popular song in a parallel world’s contest as it regresses to the mean (
1000000 - (500000 + (500000 * 0.93))). (As I noted the other day, even in very small samples you need extremely high correlations to guarantee double-maxes or similar properties, once you move beyond means; our intuitions don’t realize just what an extreme demand we make when we assume that, say, J.K. Rowling must be a very popular successful writer in most worlds simply because she’s a billionaire in this world, despite how many millions of people are writing fiction and competing with her. Realistically, she would be a minor but respected author who might or might not’ve finished out her HP series as sales flagged for multi-volume series; sort of like her crime novels published pseudonymously.)Then toss in the undoubtedly <<1 correlation between popularity and any ‘quality’… It is indeed no surprise that, out of the millions and millions of chefs over time, the best chefs in the world are not the most popular YouTube chefs. Another example of ‘the tails comes apart’ at the extremes and why order statistics is counterintuitive.
* They also report a rho=0.52 from some other experiments, which are arguably now more relevant than the 0.93 estimate. Obviously, if you use 0.52 instead, my point gets much much stronger: then, out of a million, you regress from #1 to #240,000!
A related thing that comes to mind is looking at Hacker News submissions, and how the same post will sometimes get hundreds of upvotes, and other times only get one or two. Example.
This is fantastic. Thank you!
Cool. I was going to top-level comment a possible explanation: the more info about other people’s judgements is shared (e.g. via the internet), the stronger misinformation cascades will be, because there are more judgements you observe that are ambiguously either actual data vs. just copying other people’s judgements. With more misinformation cascades, you have more things that are “popular because they’re popular”, and “unpopular because they’re unpopular” (or, stonks and undiscovered good investments).
A different story (or maybe overlapping) is that with more information about people’s judgements, drawing on a larger pool of candidates, the worse the tradeoff becomes to do your own exploration: the cost of mental processing is roughly the same, but just copying everyone else’s opinions gets a much better result (though still maybe significantly worse than is possible).