To anyone thinking this is not random, with 42 votes in:
The p-value is 0.895 (this is the probability of seeing at least this much non-randomness, assuming a uniform distribution)
The entropy is 2.302bits instead of log(5) = 2.322bits, for 0.02bits KL-distance (this is the number of bits you lose for encoding one of these votes as if it was random)
If you think you see a pattern here, you should either see a doctor or a statistician.
Looks like we’re better at randomness than the rest of the population. If I asked random people for a random number from 1 to 10, I wouldn’t be surprised to see substantially less than 3.322 bits of entropy per number (e.g., many more than 10% of the people choosing 7).
After one month and 118 responses, I’m considering this poll closed. The results are:
1) 17% 2) 21% 3) 20% 4) 24% 5) 18%
A chi-squared test says that these results do not differ significantly from uniform random responding, with a p-value of 0.78.
The main reason why I ran this poll was because I thought it might have implications for the trickier poll above. It is interesting the option #4 was the most common response in this poll, that poll, and the gamefaqs poll which that poll was based on. #4 may seem especially random, and some respondents in the other polls may have just been trying to answer at random. But this poll ended up not providing much information about that; to test it we’d need a larger sample size, and preferably a poll where respondents did not use external sources of randomness.
Question: what’s a reasonable prior over the probability distribution of poll answers? Because I downloaded the raw data, and it says:
15
22
21
24
18
...and I’m not sure what would constitute reasonable priors for the uniform distribution hypothesis versus the “aversion toward First Answer” hypothesis versus the “aversion toward First Answer and Fifth Answer” hypothesis.
My own feelings on the matter are that if you don’t know what prior to have, compute worst-case bounds.
In this case, the model that maximizes the probability of seeing this data is that each answer is 15% likely to be 1, 22% likely to be 2, 21% likely to be 3, 24% likely to be 4, and 18% likely to be 5. We can compute the probability of seeing this data under this model, and also under the “all answers are equally likely” model, and conclude that our worst-case model makes us only 3.61 times as likely to see this data.
In particular, any other hypothesis you might have can only receive this little evidence, relative to the uniform distribution hypothesis; and I believe in close-to-uniformity enough that I’m not going to be swayed by what is fewer than 2 bits of evidence.
Your question is confused. The uniform distribution hypothesis only requires that the (assumed infinite) population picks the answers independently with equal probability. Under this hypothesis, the observed poll answers (for a fixed number of respondents) will follow a multinomial distribution with parameters (0.2, 0.2, 0.2, 0.2, 0.2). A typical realization will not have an equal number of respondents giving each answer, although asymptotically the empirical frequencies will converge to equality.
Anyways, as a Bayesian, the better question is what should my posterior belief about the response probabilities be after running the poll and updating off the answers? The canonical way to do this would be to put a Dirichlet prior over the response probabilities. By the miracle of conjugacy, your posterior distribution will itself by a (generally different) Dirichlet distribution.
By taking the expectation of indicator variables like I{”probability of First Answer under 0.2″} under the posterior, you can figure out what degree of belief you must give to statements like “respondents have an aversion toward First Answer”.
Happy this helped. The Dirichlet-multinomial model gets relatively little attention because it adds nothing really new to the beta-binomial model for polls with just two responses. It’s easy to find lots of introductory, chatty introductions to the beta-binomial like this one or this one if you want to learn more...
I needed 3 random bits (and threw out any overflow), which I got by checking whether arbitrary words or phrases I thought of had an even or odd number of letters. That’s the most random completely mental (heh) way I know of, I wonder if there are others.
Average is irrelevant. What’s relevant is the standard deviation.
Since standard deviation goes as the square root of the number of items being added, phrase length for any reasonably-sized phrase, so long as it wasn’t a line of poetry, should be pretty evenly distributed.
It’s not obvious to me that it’s unbiased. My gut feeling suspects that if I randomly chose a word it’d be more likely to have an odd than an even number of letters.
I think this would be even more interesting as “pick at random, without an external source of randomness”. Sure you can get random numbers from random.org, your computer or the seconds on your watch (a nice idee), but those just blur the effect of mind-generated random numbers.
Pick your answer to this poll at random: [pollid:39]
I used random.org to generate my answer.
But, when I submitted it, I got the following:
First Answer 0 (0%)
Second Answer 0 (0%)
Third Answer 0 (0%)
Fourth Answer 1 (2%)
Fifth Answer 0 (0%)
Total 58 (100%)
The raw data contained all the 58 rows, however. Seems like there might be a bug in the result-rendering code.
To anyone thinking this is not random, with 42 votes in:
The p-value is 0.895 (this is the probability of seeing at least this much non-randomness, assuming a uniform distribution)
The entropy is 2.302bits instead of log(5) = 2.322bits, for 0.02bits KL-distance (this is the number of bits you lose for encoding one of these votes as if it was random)
If you think you see a pattern here, you should either see a doctor or a statistician.
I wish I could see a doctor-statistician. Or at least a doctor who understood statistics.
Yvain might some day have his own practice.
Here is one: http://www.ted.com/talks/ben_goldacre_battling_bad_science.html
Looks like we’re better at randomness than the rest of the population. If I asked random people for a random number from 1 to 10, I wouldn’t be surprised to see substantially less than 3.322 bits of entropy per number (e.g., many more than 10% of the people choosing 7).
Well, it’s worth noting people seem to be trainable to choose randomly: http://dl.dropbox.com/u/85192141/1986-neuringer.pdf
Apropos of the PRNG discussion in http://blog.yunwilliamyu.net/2011/08/14/mindhack-mental-math-pseudo-random-number-generators/ for which I wrote some flashcards: http://pastebin.com/CKif0fEf
Ha. I fail at random. In my defence, the universe is probably deterministic anyway.
it’s probably not, but you’re still excused ;)
After one month and 118 responses, I’m considering this poll closed. The results are:
1) 17%
2) 21%
3) 20%
4) 24%
5) 18%
A chi-squared test says that these results do not differ significantly from uniform random responding, with a p-value of 0.78.
The main reason why I ran this poll was because I thought it might have implications for the trickier poll above. It is interesting the option #4 was the most common response in this poll, that poll, and the gamefaqs poll which that poll was based on. #4 may seem especially random, and some respondents in the other polls may have just been trying to answer at random. But this poll ended up not providing much information about that; to test it we’d need a larger sample size, and preferably a poll where respondents did not use external sources of randomness.
For convenience: http://www.random.org/ or in Bash,
echo $(($RANDOM % 5 + 1))Question: what’s a reasonable prior over the probability distribution of poll answers? Because I downloaded the raw data, and it says:
15
22
21
24
18
...and I’m not sure what would constitute reasonable priors for the uniform distribution hypothesis versus the “aversion toward First Answer” hypothesis versus the “aversion toward First Answer and Fifth Answer” hypothesis.
My own feelings on the matter are that if you don’t know what prior to have, compute worst-case bounds.
In this case, the model that maximizes the probability of seeing this data is that each answer is 15% likely to be 1, 22% likely to be 2, 21% likely to be 3, 24% likely to be 4, and 18% likely to be 5. We can compute the probability of seeing this data under this model, and also under the “all answers are equally likely” model, and conclude that our worst-case model makes us only 3.61 times as likely to see this data.
In particular, any other hypothesis you might have can only receive this little evidence, relative to the uniform distribution hypothesis; and I believe in close-to-uniformity enough that I’m not going to be swayed by what is fewer than 2 bits of evidence.
Thanks! I didn’t think of that particular brainhack—I’ll be sure to use it in the future.
Your question is confused. The uniform distribution hypothesis only requires that the (assumed infinite) population picks the answers independently with equal probability. Under this hypothesis, the observed poll answers (for a fixed number of respondents) will follow a multinomial distribution with parameters (0.2, 0.2, 0.2, 0.2, 0.2). A typical realization will not have an equal number of respondents giving each answer, although asymptotically the empirical frequencies will converge to equality.
Anyways, as a Bayesian, the better question is what should my posterior belief about the response probabilities be after running the poll and updating off the answers? The canonical way to do this would be to put a Dirichlet prior over the response probabilities. By the miracle of conjugacy, your posterior distribution will itself by a (generally different) Dirichlet distribution.
By taking the expectation of indicator variables like I{”probability of First Answer under 0.2″} under the posterior, you can figure out what degree of belief you must give to statements like “respondents have an aversion toward First Answer”.
That makes sense—I had imagined doing something similar, but I had never heard of Dirichlet priors.
Happy this helped. The Dirichlet-multinomial model gets relatively little attention because it adds nothing really new to the beta-binomial model for polls with just two responses. It’s easy to find lots of introductory, chatty introductions to the beta-binomial like this one or this one if you want to learn more...
Is (the seconds’ figure in my watch) mod 5 random enough?
I used the least significant digit on my time-remaining-to-full-charge. And ended up propping up the most populated entry.
I needed 3 random bits (and threw out any overflow), which I got by checking whether arbitrary words or phrases I thought of had an even or odd number of letters. That’s the most random completely mental (heh) way I know of, I wonder if there are others.
… you could have done it more-reliably evenly by taking the mod 5 of the phrase/word length.
Considering that the average word length in English is about five letters, I suspect that’d be quite far from being uniformly distributed.
Average is irrelevant. What’s relevant is the standard deviation.
Since standard deviation goes as the square root of the number of items being added, phrase length for any reasonably-sized phrase, so long as it wasn’t a line of poetry, should be pretty evenly distributed.
It’s not obvious to me that it’s unbiased. My gut feeling suspects that if I randomly chose a word it’d be more likely to have an odd than an even number of letters.
I think this would be even more interesting as “pick at random, without an external source of randomness”. Sure you can get random numbers from random.org, your computer or the seconds on your watch (a nice idee), but those just blur the effect of mind-generated random numbers.
I rolled 1d6, intending to reroll any 6s.