I think Berkson’s paradox is explained by Why the tails come apart, with the added comment that it seems that the explanation applies not just to the tails of the distribution but in fact to any selected band (although the effect is most extreme in the tails). Simpson’s paradox is a paradox about ratio’s and different sample sizes, showing that if you average fractions you get different results than if you add enumerators and denominators, whereas Berkson’s paradox is selection bias (from the wikipedia page: conditioning on [A or B] anti-correlates A and B).
Haha! You’re right, I spoke too soon. The graph Katja used is the exact same graph used to explain Berkson’s paradox and the pattern match hit me so hard I couldn’t resist commenting. Lesson learned: think about it for 5 minutes before commenting.
Katja’s phenomenon (KP) is not an instance of Berkson’s paradox (BP) but I can see how they would often go together. Imagine that you go down a street and pick up everyone who is particularly tastefully dressed. In this group rich people will be overrepresented simply because they more easily afford good clothes. You could naively conclude from this that wealth correlates with taste. This is KP.
THEN you administer some rigorous test of good taste to everyone in the group (whatever that means). And then you discover that now the rich people in you population are *under*performing relative to the poor. For the simple reason that in your population everyone is either rich or has good taste—all the poor ones have good tastes but not all of the rich do. From this you could naively conclude that wealth *negatively* correlates with taste. This is BP.
How would this work in real life? Maybe a casual observer is more likely to fall to KP and think rich <-> good taste while a fashion expert who works exclusively with well-dressed people will think the opposite?
It’s called Berkson’s paradox and it can be used to explain all kinds of real life observations like “why are all the handsome men I date such jerks” (http://www.slate.com/blogs/how_not_to_be_wrong/2014/06/03/berkson_s_fallacy_why_are_handsome_men_such_jerks.html) or why google discovered that being good at programming competitions negatively correlated with being good at the job.
> It’s called Berkson’s paradox
Just want to note that this appears to be the *opposite* of the phenomenon that Katja was suggesting.
Katja: do people infer that taste and wealth go together?
Berkson’s (according to Slate): people infer that handsomeness and niceness do not go together.
>Katja: do people infer that taste and wealth go together?
My weak guess is yes, but not sure.
I meant to be paraphrasing you, not asking you a question :P
Also, can anyone explain the difference between Berkson’s paradox and Simpson’s paradox?
EDIT: The explanation here is helpful: http://theconversation.com/paradoxes-of-probability-and-other-statistical-strangeness-74440.
I think Berkson’s paradox is explained by Why the tails come apart, with the added comment that it seems that the explanation applies not just to the tails of the distribution but in fact to any selected band (although the effect is most extreme in the tails). Simpson’s paradox is a paradox about ratio’s and different sample sizes, showing that if you average fractions you get different results than if you add enumerators and denominators, whereas Berkson’s paradox is selection bias (from the wikipedia page: conditioning on [A or B] anti-correlates A and B).
Haha! You’re right, I spoke too soon. The graph Katja used is the exact same graph used to explain Berkson’s paradox and the pattern match hit me so hard I couldn’t resist commenting. Lesson learned: think about it for 5 minutes before commenting.
Katja’s phenomenon (KP) is not an instance of Berkson’s paradox (BP) but I can see how they would often go together. Imagine that you go down a street and pick up everyone who is particularly tastefully dressed. In this group rich people will be overrepresented simply because they more easily afford good clothes. You could naively conclude from this that wealth correlates with taste. This is KP.
THEN you administer some rigorous test of good taste to everyone in the group (whatever that means). And then you discover that now the rich people in you population are *under*performing relative to the poor. For the simple reason that in your population everyone is either rich or has good taste—all the poor ones have good tastes but not all of the rich do. From this you could naively conclude that wealth *negatively* correlates with taste. This is BP.
How would this work in real life? Maybe a casual observer is more likely to fall to KP and think rich <-> good taste while a fashion expert who works exclusively with well-dressed people will think the opposite?