You don’t know it was a Poisson distribution (there are lots of distributions natural phenomena follow; the negative binomial and lognormal also pop up a lot in human contexts),
The Poisson distribution is the distribution that models rare independent events. Given how involved you are with prediction and statistics, I’d expect you to know that.
The Poisson distribution is the distribution that models rare independent events.
Are number of fighter pilot victories, clearly, a priori, going to be independent events? That a pilot shooting down one plane is entirely independent of whether they go on to shoot down another plane? (Think about the other two distributions I mentioned and why they might be better matches...)
Distributions are model assumptions, to be checked like any other. In fact, often they are the most important and questionable assumption made in a model, which determines the conclusion; a LW example of this is Karnofsky’s statistical argument against funding existential risk, which driven entirely by the chosen distribution. As the quote goes: ‘they strain at the gnat of the prior who swallow the camel of the likelihood function’.
I personally find choice of distribution to be dangerous, which is why (when not too much more work) in my own analyses I try to use nonparametric methods: Mann-Whitney u-tests rather than t-tests, bootstraps, and at least look at graphs of histograms or residuals while I’m doing my main analysis. Distributions are not always as one expects. To give an example involving the Poisson: I was doing a little Hacker News voting experiment. One might think that a Poisson would be a perfect fit for distribution of scores—lots of voters, each one only votes on a few links out of the thousands submitted each day, they’re different voters, and votes are positive count data. One would be wrong, since while a Poisson fits better than, say, a normal, it’s grossly wrong about outliers; what actually fits much better is a mixture distribution of at least 3 sub-distributions of Poissons and possibly normals or others. (My best guess is that this mixture distribution is caused by HN’s segmented site design leading to odd dynamics in voting: the first distribution corresponds to low-scoring submissions which spend all their time on /newest, and the rest to various subpopulations of submissions which make it to the main page—although I’m not sure why there are more than 1 of those).
So no, I hope it is because of, rather than despite, my involvement with stats that I object to Rolf’s casual assumption of a particular distribution to create a fully general counterargument to explain away data he has not seen but dislikes.
In particular notice that any deviations from Poisson are going to be in the direction that makes Rolf’s argument even stronger.
No, they’re not, not without even more baseless assumptions. The Poisson is not well-justified, and it’s not even conservative for Rolf’s argument. If there was a selection process in which the best pilots get to combat the most (a shocking proposition, I realize); then many more would cross the threshold of at least 1 kill than would be predicted if one incorrectly modeled kill rates as Poissons with averages. This is the sort of thing (multiple consecutive factors) which would generate other possible distributions like the lognormal, which appear all the time in human performance data like scientific publications. (‘...who swallow the camel of the likelihood function’.)
And this still doesn’t address my point that you cannot write off data you have not seen with a fully general counterargument—without very good reasons which Rolf has not done anything remotely like showing. You do not know whether that extremely low quoted rate is exactly what one would expect from pilots doing their level best to kill others without doing a lot more work to verify that a Poisson fits, what the rate parameter is, and what the distribution of pilot differences looks like; the final kill rate of pilots, just like soldiers, is the joint result of many things.
The Poisson distribution is the distribution that models rare independent events. Given how involved you are with prediction and statistics, I’d expect you to know that.
Are number of fighter pilot victories, clearly, a priori, going to be independent events? That a pilot shooting down one plane is entirely independent of whether they go on to shoot down another plane? (Think about the other two distributions I mentioned and why they might be better matches...)
Distributions are model assumptions, to be checked like any other. In fact, often they are the most important and questionable assumption made in a model, which determines the conclusion; a LW example of this is Karnofsky’s statistical argument against funding existential risk, which driven entirely by the chosen distribution. As the quote goes: ‘they strain at the gnat of the prior who swallow the camel of the likelihood function’.
I personally find choice of distribution to be dangerous, which is why (when not too much more work) in my own analyses I try to use nonparametric methods: Mann-Whitney u-tests rather than t-tests, bootstraps, and at least look at graphs of histograms or residuals while I’m doing my main analysis. Distributions are not always as one expects. To give an example involving the Poisson: I was doing a little Hacker News voting experiment. One might think that a Poisson would be a perfect fit for distribution of scores—lots of voters, each one only votes on a few links out of the thousands submitted each day, they’re different voters, and votes are positive count data. One would be wrong, since while a Poisson fits better than, say, a normal, it’s grossly wrong about outliers; what actually fits much better is a mixture distribution of at least 3 sub-distributions of Poissons and possibly normals or others. (My best guess is that this mixture distribution is caused by HN’s segmented site design leading to odd dynamics in voting: the first distribution corresponds to low-scoring submissions which spend all their time on /newest, and the rest to various subpopulations of submissions which make it to the main page—although I’m not sure why there are more than 1 of those).
So no, I hope it is because of, rather than despite, my involvement with stats that I object to Rolf’s casual assumption of a particular distribution to create a fully general counterargument to explain away data he has not seen but dislikes.
Rolf addressed that point:
In particular notice that any deviations from Poisson are going to be in the direction that makes Rolf’s argument even stronger.
No, they’re not, not without even more baseless assumptions. The Poisson is not well-justified, and it’s not even conservative for Rolf’s argument. If there was a selection process in which the best pilots get to combat the most (a shocking proposition, I realize); then many more would cross the threshold of at least 1 kill than would be predicted if one incorrectly modeled kill rates as Poissons with averages. This is the sort of thing (multiple consecutive factors) which would generate other possible distributions like the lognormal, which appear all the time in human performance data like scientific publications. (‘...who swallow the camel of the likelihood function’.)
And this still doesn’t address my point that you cannot write off data you have not seen with a fully general counterargument—without very good reasons which Rolf has not done anything remotely like showing. You do not know whether that extremely low quoted rate is exactly what one would expect from pilots doing their level best to kill others without doing a lot more work to verify that a Poisson fits, what the rate parameter is, and what the distribution of pilot differences looks like; the final kill rate of pilots, just like soldiers, is the joint result of many things.