Yes, “four times as likely” is not the same as an odds ratio of four. And the problem here is the same as the problem in army1987′s LL link that odds ratios get mangled in transmission.
But I like odds ratios. In the limit of small probability, odds ratios are the same as “times as likely.” But there’s nothing 4x as likely as 50%. Does that mean that 50% is very similar to all larger probabilities? Odds ratios are unchanged (or inverted) by taking complements: 4% to 1% is an odds ratio of about 4; 99% to 96% is also 4 (actually 4.1 in both cases). Complementation is exactly what’s going on here. The drug companies get 1.2x-1.3x more positive results than the independent studies. That doesn’t sound so big, but everyone is likely to get positive results. If we speak in terms of negative results, the independent studies are 2-3x likely to get negative results as the drug companies. Now it sounds like a big effect.
Odds ratios give a canonical distance between probabilities that doesn’t let people cherry-pick between 34% more positives and 3x more negatives. They give us a way to compare any two probabilities that is the obvious one for very small probabilities and is related to the obvious one for very large probabilities. The cost of interpolating between the ends is that they are confusing in the middle. In particular, this “3x more negatives” turns into an odds ratio of 4.
Sometimes 50% really is similar to all larger probabilities. Sometimes you have a specific view on things and should use that, rather than the off the shelf odd ratio. But that doesn’t seem to be true here.
Thank you for this. I’ve always been frustrated with odds ratios, but somehow it never occurred to me that they have the beautiful and useful property you describe.
I don’t know as much about odds ratios as I would like to, but you’ve convinced me that they’re something I should learn thoroughly, ASAP. Does anybody have a link to a good explanation of them?
Yes, “four times as likely” is not the same as an odds ratio of four. And the problem here is the same as the problem in army1987′s LL link that odds ratios get mangled in transmission.
But I like odds ratios. In the limit of small probability, odds ratios are the same as “times as likely.” But there’s nothing 4x as likely as 50%. Does that mean that 50% is very similar to all larger probabilities? Odds ratios are unchanged (or inverted) by taking complements: 4% to 1% is an odds ratio of about 4; 99% to 96% is also 4 (actually 4.1 in both cases). Complementation is exactly what’s going on here. The drug companies get 1.2x-1.3x more positive results than the independent studies. That doesn’t sound so big, but everyone is likely to get positive results. If we speak in terms of negative results, the independent studies are 2-3x likely to get negative results as the drug companies. Now it sounds like a big effect.
Odds ratios give a canonical distance between probabilities that doesn’t let people cherry-pick between 34% more positives and 3x more negatives. They give us a way to compare any two probabilities that is the obvious one for very small probabilities and is related to the obvious one for very large probabilities. The cost of interpolating between the ends is that they are confusing in the middle. In particular, this “3x more negatives” turns into an odds ratio of 4.
Sometimes 50% really is similar to all larger probabilities. Sometimes you have a specific view on things and should use that, rather than the off the shelf odd ratio. But that doesn’t seem to be true here.
Thank you for this. I’ve always been frustrated with odds ratios, but somehow it never occurred to me that they have the beautiful and useful property you describe.
I don’t know as much about odds ratios as I would like to, but you’ve convinced me that they’re something I should learn thoroughly, ASAP. Does anybody have a link to a good explanation of them?
http://lesswrong.com/lw/8lr/logodds_or_logits/ would be helpful for you, I think, since an explanation/introduction was the stated goal.
Sorry, I don’t have any sources. If you want suggestions from other people, you should try the open thread.
Some related words that may be helpful in searching for material are logit and logistic (regression).